Siverchair Information Systems
Perhaps this project is the Crown of my Software Develoment experiences
ONE PLATFORM FOR SCHOLARLY PUBLISHERS
Silverchair integrates and delivers scholarly and professional content from a single platform – journals, books, video, custom formats, and more.
Siverchair is perhaps the greatest achievement I have ever realized in my Software Development career. Of course, there is the prestige of working for the Department of Interior and the BIA, both of which thrust me into the hallowed halls of the Federal Government and answering to the Secretary of the Interior, Bruce Babbitt, but as a personal experience, Silverchair takes the crown.
I have never worked with a more dedicated and professional group of people bar none. And the only reason I left Silverchair is because I did not want to become an employee. They made me some very attractive offers, but I am an independent contractor to my very bones. After the 18 month engagement with them, I had to leave or risk falling into that IRS shadow land of a statutory employee.
I am going to depart from the usual past client format here because I really want to talk about this engagement in a story telling mode to illustrate how a large complex project should be managed and the Silverchair team that really stepped up to the plate to create Silverchair’s SCM-6 platform.
I interviewed twice for the position at Silverchair. Though the first interview went well, I was at the time too pricey for their budget. I assume they wanted me to be involved with updating their aging SCM-5 platform because there was no mention at the time of a new platform to be developed. They called me several weeks later wanting me to interview with a different team that might be interested in my services. This is where I met Holly who was to become the product manager of the SCM-6 platform and Lauran who was to be one of the ‘Architects’ of the new platform. The interview was done around November 10, I was on premises by the end of that week.
After I completed the normal routine of setting up my workspace I was introduced to the current working team for SCM-6. We were not housed in the main Silverchair building, instead we where located in a leased area next door. The team consisted of five people besides myself. The discussion revolved around what we were planning to build and how we were going to put together a general architecture that a larger team would be implementing.
My actual participation began when Lauren had me log onto the source control platform and download the project. We were using SVN and Jira as our process tools. I opened up the SCM-6 project, and it consisted of one class and a SQL-Database with no entities yet defined. I asked Lauren how she expected me to proceed and here response was, You figure it out, this part of SCM-6 is yours to do as you see fit. I sat there for a while thinking about just packing up and leaving. From our earlier discussions I had a grasp of what the final product was supposed to be and the 18-24-month period from which to get it done. Lauren and Josh, the other architect in the Architected team where more consumed with the ZipLine product which would be the Publishing Article input engine as I understood it.
I guess I had expected some sort of design document to guide me, but there was none. Just a project with one class consisting of a namespace and a class declaration. No other code at all. And like I said, a SQL-Server database with no tables. I went around talking to the BA, the Product manager and Josh and Lauren for more information gathering up notes. I was relieved when the day ended and I could go home. I moaned and groaned to my wife about how this was another Rudd Lighting type of project and how I wanted to just excuse myself from it. But my wife hit it on the nail head when she said, “don’t you see, they have handed this project over to you to design it as you like. You are in charge of how this project develops!”
She was right, while nobody said so in as many words and nobody officially told me I was to be the Architect of SCM-6, there is was in my lap with an apparent promise of no interference. I did have a set of features and user stories that Holly, Lauren and others had developed so I had somewhat of a clue as to what was expected. I knew that I would divide the project into three distinct pieces. Th UI, the Business Model and the Repository. I had been told in no uncertain terms was I to use the Microsoft ASP-MVC platform, this would be an ASP-WinForms application using VB.Net. Okay I can work with this I said.
I decided on a set of patterns that I though would work well with this endeavor, chief among them was the Façade pattern. The UI, or ASP.Net portion of this application would have absolutely no knowledge of the SCM-6 Object Model inner workings. It would make simple requests and get simple responses. There would be a set of data model interfaces that would be shared across all three areas, but that was it. I called the business tier the Object Model (‘a mistake that caused others to misunderstand some key concepts’) because the entire Business layer would be exposed from a single class that exposed sub classes that exposed methods that would return data models only. Almost all of the method signatures would have a single integer argument.
The central idea in this was that absolutely no business-related exceptions would ever reach the UI part of the application. You either got data returned or you got a null return. The web site would handle the null returns in a way that gave the user options based upon the results of a request. Exceptions would be logged into a database that tracked these exceptions and would be remedied in the background. The users did not need to know the details of these exceptions.
I had wanted to develop a repository that directly access the data store through ADO.Net, but in a later discussion with an expanded team I was overruled with a decision to use Microsoft’s Enterprise Data Access Layer. I was not happy with this, but in the end, it turned out to be a satisfactory choice.
The key requirement of this application was speed, something they were having problems with in the SCM-5 platform. And I already knew that the speed issue was going to be in the retrieving of data from the datastore. I saw the MS-DAL as a liability here, but in working with Jay, the person who strongly advocated using the MS-DAL we were able to make customizations that minimized some of the DAL’s shortcomings. And yet (IMHO) , we were not getting the performance we needed from the database.
I knew what the issue was, but how was I going to manage the cure in such a large project and with several scrum teams that were going to have their own opinions on the subject. Let me tell you, managing programmers is like trying to herd cats, it is a very difficult task.
The problem was in how results where returned from tables or views. The standard practice in almost all cases was to take a filed by name and populate a model’s property with that field. This required some background steps that most programmers don’t know about, the unboxing of strings, doing a lookup to get the index of the field in the result set, then finally transferring the data from the result set to the model. We are talking about milliseconds here, not a lot of time or so it would seem. But if a proves takes 100 milliseconds times how many fields times how many rows those milliseconds add up.
I knew a way to reduce this time by what I thought would be upwards of a 50-60% percent boost in performance. That would be huge. “As a side note to this story, my solution seemed very anal retentive to several programmers because it required more work from them than the good ole method of using strings to identify fields. So, they decided to time test my solution against the standard way of doing things and lo and behold they were right, the were not getting the 50-60% boost in performance. They were measuring a 90% performance boost. The complaints on this issue evaporated”
My solution involved referring to fields by their index number in the result set rather than the field name. To make this easy for developers, I introduced the idea of using Enums that represented the string names, but were references to the actual index position of a given result set. If you are in any way a programmer you will instantly see the developing nightmare in maintaining this strategy as tables change during development, Views are even more subject to changes in result sets, and result sets from Stored Procedures are perhaps the worst place to try and maintain such a scheme. That is why, while many architects know this index vs string dilemma few will attempt it because of the maintenance problem.
Well there was also a cure to this maintenance nightmare that automated the creation of the enums during every build to ensure that the enums where always in tune with the result sets being returned from the data layer. It’s called the T4 Code engine, A little known and used tool in the Microsoft arsenal of programmer tools. I knew about it because I had been involved in the early days of developing the Microsoft Domain Specific Language (DSL) initiative of which the T4 engine was a component of. During Compile time, it uses a Scripting language (VB or C#) to develop code files using templates and data sources. What I came up with, was having the templates have a complete list of all result sets to be returned by the repository, read their return fields, calculate the index position, then create the enums needed to access them. This made the entire Indexing scheme palatable to the programmers, even the ones that had to write the T4 scripts. As we did trial runs in comparison to the earlier SCM-5 platform, the difference was breathtaking. I have to admit that part of the performance enhancement was also due to another architect that developed the Semantic Search platform for SCM-6, it was a brilliant use of decision trees.
Along the line there were some deviations from the architecture that I had to chase down. Some developers where breaking the facade pattern by introducing Lambda arguments into the method calls of the Object Model. These in return generated exceptions on the UI portion of the project. Also, some folks where introducing Business Tier level HTML creation along with its associated references to the Web libraries. This would tie the Business layer to this version of ASP.Net, and require the Zipline application to have some possible associations with the web application. I guess in some eyes I was like a Captain Willian Bligh in their eyes chasing down these violations of the architecture , oh no here he comes with another lecture on separation of concerns and the like.
I remember in one instance, I was asked by another architect to not be so problematic about using lambdas in the object model arguments. He was telling me that it was OK that business level exceptions appeared on the UI side of things, we’ll just explain to the user that there were some problems that need fixing. Just as we were having that discussion, the IT director, Scott came up to us and complained that he wished there was a way we could prevent UI side exceptions. When Scott left the conversation we just eyed each other and that was the end of the “don’t be so strict’ discussion.
Midway through the project, I published an Architectural document explaining the entire architecture of SCM-6 to management and of course all of the development staff including the BA’s and Product/Project managers. Believe it or not, very few read it. I am not surprised, it was not a dime store novel with exciting mystique, romance and mystery. It was a dry manifest with some illustrations. It was not until some meeting a few weeks later during a meeting that the BA on my team screamed with excitement, “I got it, I got it.” She had finally grasped the architecture in its entirety.
In the end, the project was a raging success. Sure, I like to think that the SCM-6 architecture had a lot to do with that success. But in reality, it was the commitment of the Executive management that had the foresight to implement solid software development processes. They paid for all of their employees and their contractors to attend a course on Agile Development. They adopted SVC as their source control tool and Jira as their project management tool. All excellent tools. And most of all, they did no get in the way. They trusted their staff to get the job done and they did. Everyone stepped up to the plate because they were expected to through mutual respect and trust. I hated leaving Silverchair. I met some very competent and personable people there. They had definitely influenced my life’s outlook more than they can imagine.
On my last day there, I was busy doing technology transfer to some folks that were taking over my role. It was courteous and professional, but that I was leaving this day changed things and changed my relationships with everyone. I was not part of the team anymore. That’s how things usually go for a contractor. On this last day some good byes were said, hands shaken and one last attempt by Scott to lure me into being an employee. I said no and that was that. We were all parting ways. I felt a little disappointed that there wasn’t more to this than just walking out the door on this last day. Then the Silverchair monsters proceeded to crush me as a left my desk, the entire staff, including the corporate executives gathered together and gave me a standing ovation. My jaw was on the ground and my eyes where full of water…I had never experienced anything like this in my entire life.
That is why I am doing this Client History in a long essay style, to say thanks to all of them and to let all of my readers know, that Silverchair was a truly magical place to work!
As a last note, Josh, my direst supervisor at Silverchair was asked to fill out a reference form from a potential client. He sent me a copy of his response; here it is:
Sent: Wednesday, April 23, 2014 2:15 PM
To: Jamila Malluf <firstname.lastname@example.org>
Subject: Fwd: Josh Supervisor Reference for Ibrahim Malluf
I sent this email off to Tessa for the reference. Hope y'all are having a great day!
---------- Forwarded message ----------
From: Josh <email@example.com>
Subject: Re: Josh Supervisor Reference for Ibrahim Malluf
My responses are in-line below,
On Tue, Apr 22, 2014 at 8:44 PM, T Michaels wrote:
Ibrahim Malluf has provided you as a reference to ..... I recognize you are busy but could you please briefly answer the below questions? Please email with any questions or let me know if you prefer to speak on the phone.
What was the professional relationship with the candidate? At what company did you work together?
I worked with Ibrahim at Silverchair Information Systems, a medical publishing provider in Charlottesville, VA. I was the Enterprise Architect for a new project to build a new journal publishing platform (a very tall order) and Ibrahim was a contractor that we hired, as he came highly recommended, and did exceptionally well in our (very technical) interviews.
Did they report to you? How long did you work together? Was he/she working on site or remotely.
Yes, Ibrahim worked on the team under myself and another architect, and we all worked very closely. We worked directly together for about a year, and on related teams for a bit longer than that. Ibrahim's time was primarily on site.
What job did the candidate perform? Please describe his responsibilities.
Very soon after hiring Ibrahim, we asked him to manage a team of developers, and he was responsible for understanding our architectural vision, often doing architectural/software design work himself, and communicating that to his team, as well as making sure his team was staying on course and helping to mentor them technically through some of the more difficult development. Over time he provided excellent technical leadership across numerous demanding tasks and projects.
If applicable, why did the candidate leave this position?
Ibrahim worked on several projects while at Silverchair, but ultimately (after several significant extensions of his contract) the contract expired, and he moved on.
What can you tell me about his/her knowledge of the job?
We relied very heavily on Ibrahim's technical excellence and experience. He brought a very wise balance of practicality and initiative in what can only be described as a chaotic environment. His deep knowledge and experience helped bring practical solutions to tough problems on a regular basis.
What can you tell me about the candidate's work:
I only wish more people had his ethic. He took ownership of problems and was very invested in seeing them solved.
We relied on his initiative, and were very rewarded for doing so!
(c) Attendance & Punctuality?
We never had any problems in this regard.
(d) Communication Skills?
Part of Ibrahim's responsibilities was communicating directly with project managers and architects, and translating that for developers under his management. He did a great job.
What can you tell me about the candidate's technical knowledge on the job, especially responsibilities for DNN (DotNetNuke), C#, .NET?
For this project we did not use DNN or C#, but VB.NET, Microsoft SQL Server, ASP.NET, AppFabric, and a slew of other technologies. Again, his expertise and experience with these technologies was indispensable, and his ability to learn new technologies (primarily AppFabric, if I remember correctly) was vitally important.
What were the candidate's professional/technical weak points?
It's sort of hard to answer this question. We were all using at least some technologies that were new to us, and there were some that had to be picked up "on the run" under very tight deadlines. Ibrahim learned what he needed to learn on the (very demanding!) time frame he needed to learn them... so to me, as his manager, that counted a lot more as a strength than a weakness.
How did the candidate react to pressure and/or deadlines?
He got the work done, and was passionate about doing it well. So, I'd say he reacted well.
Why and how was the candidate hired and/or promoted into this role?
Our company was advertising for contracting and contract-to-hire positions for this project, and he responded, had excellent references, and did very well in our "technical grilling" which we engaged in to a very high degree for this demanding project.
Were there any specific successes or projects the candidate was responsible for or was engaged in?
There were many! This was a huge project with many moving parts, and Ibrahim took a direct role in helping on numerous fronts. His work was acknowledged to the point that in one of our company-wide meetings, the executive team took a moment to speak in recognition of his work, and he received a standing ovation of applause from the entire company, many of whom were directly affected by his work. I'm happy to go into deeper detail, but that might be easier to do in a phone conversation than in email bullet points!
How did the candidate get along with peers, supervisors and subordinates?
He got along well. All that worked closely with him held a deep respect and appreciation for him. With some he came across as vocally opinionated... which was true, but was a result of his passion for doing things well, and avoiding mistakes.
Would you hire this person again?
I hope that this email has helped! I am very willing to discuss this in a phone conversation. It's probably easiest to schedule a time to have a phone call, as I fear trying to "catch" each other on the phone will lead to games of phone tag... but I will be available tonight after about 6pm at 434-260-1171, and I'm happy to set up a specific meeting time during business hours on Thursday or Friday if that works.