Contractors Who Built Website Blame Each Other For All The Problems

from the nice-try dept

With all the problems associated with the rollout, a bunch of fingers (including ours) pointed at the usual list of government contracting cronies who built the thing. The deal was done under an existing contract (so no open bidding) and involved the same “usual suspects” who have been connected to a number of other large government computer systems debacles. Not included anywhere in the list were companies with experience building large-scale web services — which you’d think would be helpful here. However, in testimony before Congress, the contractors are insisting that it’s not their fault. CGI Federal was the main contractor behind the site, and Cheryl Campbell, a senior VP from the company, is in charge of trying to point fingers elsewhere, mainly at the Centers for Medicare and Medicaid Services (CMS), which CGI Federal says was in charge of the actual building of the site.

CMS serves the important role of systems integrator or “quarterback” on this project and is the ultimate responsible party for the end-to-end performance of the overall Federal Exchange.

Basically: it’s the government’s fault. We just build the damn thing. If they didn’t tell us to build the right thing, or test it properly, well, it’s their fault. Also, someone else we won’t name is really at fault:

Another contractor was awarded the contract for the Data Services Hub portion of the Federal Exchange.

Oh, and also another unnamed contractor:

The first set of issues for users dealt with the enterprise identity management (or EIDM) function provided by another contractor, which allows users to create secure accounts.

Of course, it’s not too difficult to figure out who the “other” contractor is. Because it’s on the panel too. QSSI built the Data Services Hub and the “EIDM” functions mentioned, and QSSI is owned by Optum, whose executive vice president Andy Slavitt is testifying as well. And, you know, it’s not his fault. First, he insists that the Data Service Hub worked splendidly throughout, no matter what anyone else might say. EIDM, of course, is having some trouble, but that? Why, other vendors are to blame there too:

It is relevant to note that the EIDM tool is only one piece of the federal marketplace’s registration and access management system, which involves multiple vendors and pieces of technology. While the EIDM plays an important role in the registration system, tools developed by other vendors handle critical functions such as the user interface, the e-mail that is sent to the user to confirm registration, the link that the user clicks on to activate the account, and the web page the user lands on. All these tools must work together seamlessly to ensure smooth registration

In other words, if only those other vendors did their job right, the whole thing would work much better. Oh yes, also someone (nameless) decided to change the specs at some late date:

It appears that one of the reasons for the high concurrent volume at the registration system was a late decision requiring consumers to register for an account before they could browse for insurance products. This may have driven higher simultaneous usage of the registration system that wouldn’t have occurred if consumers could “window shop” anonymously.

The final note, going back to CGI Federal, is to remind Congress that building websites is really hard.

Unfortunately, in systems this complex with so many concurrent users, it is not unusual to discover problems that need to be addressed once the software goes into a live production environment. This is true regardless of the level of formal end-to-end performance testing — no amount of testing within reasonable time limits can adequately replicate a live environment of this nature.

That’s true to some extent, but it doesn’t excuse many, many of the overall problems with the system, which did not appear to be built with any recognition of how to build a high-traffic transactional website. While CGI Federal would like to point fingers at everyone else, it was its name on the contract, which it received through questionable means, and it should take at least some responsibility for it. Perhaps, if it was so “complex,” it shouldn’t have taken on the job.

Filed Under: , ,
Companies: cgi federal, optum, qssi

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Contractors Who Built Website Blame Each Other For All The Problems”

Subscribe: RSS Leave a comment
That One Guy (profile) says:

You know, if it’s really that hard for them to build a working site, sounds like it might be time to give the job to some other company, or at least put the offer of such on the table. I bet the possibility of losing a multimillion dollar contract, potentially multiple such contracts in the future, would get them to shape up quite fast.

btr1701 (profile) says:

Re: Re:

You know, if it’s really that hard for them
> to build a working site, sounds like it might
> be time to give the job to some other company,
> or at least put the offer of such on the table.

Someone (or multiple someones) need to go to jail over this whole thing, not just lose a contract. Instead, the same people who already flushed a half-billion dollars of our tax money down the crapper are going to get even more money to fix it. And the government bureacrats in charge, like Sibelius, instead of losing their jobs, apparently “have the full confidence of the president”, at least they do according to his spokeshole, Jay Carney.

And they really need to stop trying to minimize their gross incompetence by calling the problems with the Obamacare website “glitches” and “technical snags”.

A complete catastrophic failure of a system built on 10-year-old outdated web tech is not a “glitch” under any commonly accepted definition of that term.

Anonymous Coward says:

This could have easily been avoided -- at very low cost

There are people who support this and people who don’t.

Among the people who support it are numerous programmers, testers, system analysts, system admins, network architects, etc.

It would have been much cheaper to simply have the feds manage the project, open-source the entire thing, and let those who support it perform the development.

Yes, yes, yes, I know that has its drawbacks: but this cost a frigging fortune and IT DOESN’T WORK. How much worse could it possibly be? And not only that: this monstrosity isn’t open-source, therefore it’s (a) insecure and (b) insecurable. (I trust everyone is aware that security by obscurity never works, therefore closed-source code is insecure by its very nature. Anyone who isn’t aware should undertake remedial education in security fundamentals.)

Some very impressive projects have been built in this manner — and many of them have demonstrated the ability to scale radically when required.

madasahatter (profile) says:

Re: This could have easily been avoided -- at very low cost

The advantage Open Source is that anyone can contribute security patches not just the internal security team. Thus “all bugs are shallow” fits. The problem with closed source is the patch team has a finite amount of time a probably plenty of bugs to patch. Some must be delayed because of time constraints and inevitably judgment errors will be made.

Your point about all code having errors, bugs, and security mistakes is true for any reasonably large project. Open Source says help us make the code better by inviting others to help and submit code patches.

Anonymous Coward says:

Re: This could have easily been avoided -- at very low cost

yes, lets make it into a ‘open-source’ Vs. ‘proprietary’ argument, that’s they way!!!

when we ALL KNOW, it’s a Democrat Vs. Republican argument, Republicans are due for a win, ‘anytime now’.

Its not like the republicans are skill at picking fights they can win or anything !

At least Mr Masnick makes ONE person supporting them, again, they cant even agree on anything amongst themselves.

Anonymous Coward says:

Re: Re: This could have easily been avoided -- at very low cost

This is part of the problem. You can’t even have a discussion about how the rollout should have been run without asshats running in and screaming about how it’s a partisan debate. I’m sorry that you feel this particular example of government incompetence is a black mark for your particular party but that doesn’t really speak to anyone else’s party affiliation (or lack thereof) nor does it address the issues raised.

Anonymous Coward says:

Re: This could have easily been avoided -- at very low cost

There is nearly no way that the number of developers needed to develop this site, would donate their time to an open source project to do so.

Plus open sourcing the software is only part of the battle. This system needs to push/pull information from multiple databases (not open source btw, loaded with personal information) from multiple agencies. That is an integration problem that open source would not have been able to solve.

I hadn’t actually heard yet that this was a no-bid contract. That was a serious error. This project was too big in scope to be slid in under another contract. That is a key driver in how and why its failing spectacularly. It was a huge development effort, but from the contract side is was viewed as a trivial add on.

I, know, its shocking to refer to a $400 million project as trivial, but that seems to be how the contractors treated its development and integration.

John Fenderson (profile) says:

Re: Re: This could have easily been avoided -- at very low cost

That is an integration problem that open source would not have been able to solve.

Open sourcing this would certainly not have guaranteed success, and the code itself is only one small part of the scope of the project, but… the code portion of this is certainly something that can be handled through the OSS model. Similar problems, on a similar scale, are handled with OSS code every day.

Personally, the problem with this project has nothing whatsoever to do with whether the code is open source or not, and everything to do with incompetence on the part of the contractors. Open sourcing would not have affected that.

Anonymous Coward says:

Re: Re: This could have easily been avoided -- at very low cost

“There is nearly no way that the number of developers needed to develop this site, would donate their time to an open source project to do so.”

Nobody is saying they should donate their time. Ask all the Red Hat employees.

Learn the difference between free speech and free beer.

Rocco Maglio (profile) says:

Integrator failure

When you have a large failure like the ACA web site, the blame starts with the integrator which was Centers for Medicare and Medicaid Services (CMS). They are the ones that split up the work to the contractors, identify requirement, and accept the finished software. If they fail at this there is no way the site will work properly. The contractors probably did not do a good job here, but even if they had the project would still fail with bad system integration.

madasahatter (profile) says:

Re: Integrator failure

I read there are 55 contractors involved plus numerous preexisting systems. Something as simple as basic the data format for an individual’s personal information must be specified in gory detail so everyone knows what it is. From the comments the insurance companies about garbled data, it appears even this basic information does not have a standard format within the system.

vastrightwing (profile) says:


1 Companies competent at implementing viable systems are not good at procuring government contracts.

2 There is no accountability: no one cares if it doesn’t work. The contractor still gets paid.

3 The contractors built what was on the spec. It’s not their fault the spec was wrong and incomplete.

4 Obama-care is a tax, not a viable health care benefit. The government will get their tax.

Anonymous Coward says:

Having worked in construction for 40 years I can assure the following statements are true:

1. You can not build it right if it is not designed right.

(Constructions firms do the building; design firms do the designing and rarely are the two the same firm.)

2. Design firms can not design it right if the design specifications are wrong.

(Design firms do the designing; project management firms make the specifications and rarely are the two the same.)

3. Project Management firms can not write the correct specifications if the owners do not know what they want and differences of opinion of what it is they want.

(The owners of the health care system is the American people.)

It is impossible to make a system work when there is no conscientious on what it is the system is suppose to do.

If one wants to understand how big a debacle the health care situation is one really needs to return to another era around 1900 and recall how Americans viewed alcohol.

Across the nation there were saloons. Saloons were not like present days bars. There was alcohol, gambling, and whores. Killings, robberies, and sexual disease were common combined with a living standard so low that some of today’s more repugnant areas of degenerated third world poverty stricken cities would appear prosperous by comparison.

As this unsavory section of town was unacceptable to be elitists of the day a reform movement was launched. The 18th was passed ushering in the most violent ear of American existence since the bloody days of the Civil War. Streets of major cities rain red with blood from mob hits.

The debacle formally ended with the passage of the 21th amendment.

More relevant the Republican is still hated for the elitist know it all that produced the 18th amendment and the single minded psychology of religious fundamentalists controlling government.

This is not going to end well for the Democrats.

But it is going to take some time. It took 65 years plus for the Republicans to reach the depth of hatred they had in the 1930 and they still have not recovered anywhere near their status of 1900.

Also, health care will be just as completely different in the future as alcoholic and saloons are today (from 1900).

Anonymous Coward says:

Re: Re:

Across the nation there were saloons. Saloons were not like present days bars. There was alcohol, gambling, and whores. Killings, robberies, and sexual disease were common combined with a living standard so low that some of today’s more repugnant areas of degenerated third world poverty stricken cities would appear prosperous by comparison.

Guess you’ve never been to a Bikers Bar.

Steve R. (profile) says:

Its the "free" money, not the product

Was the contract awarded as a political reward?

This may be similar to Solyndra where a company gets a so-called “loan” not to actually produce anything useful, but to get “free” taxpayer $$$$.

Since the website developers may have been paid for “work” already done, will bankruptcy be the next step?

ArkieGuy (profile) says:

A different perspective

I’m a programmer that has worked for a CMS subcontractor for many years (no, not one of the ones involved with this system) and I have to say that the comment that CMS is (at least partly) to blame may be dead on.

CMS is notorious for requirements that leave out most of the details. Imagine that you raise goats in your back yard and need a fence to keep them in. If CMS were writing the RFP (request for proposal), it might ask for “a fence” and when pushed to specify what kind of fence, they might answer “we really like picket fences”. Then after you quote them a picket fence and start building it, someone at CMS would ask… “Ummm… will this keep the goats in?”. And when you ask “why didn’t you put that in the requirements”, you are answered with “you should have just known”….

And no, I’m NOT making this up – I’ve been in the meetings where this exact reasoning was used.

Oh, and I do know that on this particular project at the state level, requirements that were needed to hit the MANDATED Oct 1 implementation date weren’t finalized on Oct. 1 by CMS. Of course it’s not all CMS’s fault – they have to do what congress tells them to do even if it’s impossible (in many cases congress sets the implementation date without any input from the folks that are going to implement it).

John Fenderson (profile) says:

Re: A different perspective

you are answered with “you should have just known”

This problem isn’t limited to CMS (or even the software industry) by a long shot. It’s everywhere.

A part of the problem is with the contractors who accept these vague requirements and offer up bids on them without actually knowing what the work entails. An RFP that just says “build a fence (we like picket)” is obviously trouble to begin with and shouldn’t be bid on without further clarification.

Any contractors willing to bid on something like that are, themselves, being irresponsible in that it is literally impossible to know what to bid. Good contractors would just chuckle and throw the RFP in the waste basket.

This disaster is the result of problems in the industry across the board. Despite what people seem to think (this is an example of government screwing things up), this is actually something that happens frequently in the purely private sector as well. It’s just easy to ignore it until it happens to a multimillion dollar project.

DannyB (profile) says:

A suitable contactor must exist somewhere

Surely there is a company that knows how to build web applications that has built them on a massive scale and garnered the hatred of copyright maximalists everywhere.

Oh, that’s why the government wouldn’t want to talk to them.

Or vice versa. Maybe they don’t want to work on a job mired down in bureaucracy and politicians.

Alt0 says:

Welcome to launch day!

One source the government would never consider (but would most likely do the best job) is the video game industry.

They build entire worlds that get populated by MILLIONS of players in A SINGLE DAY! (all of whom need to register and set up payment)

OK there’s some lag, maybe a que to wait in a while, but most have it down to almost a science these days.

Andrew D. Todd (user link) says:

All's Well That Ends Well.

As far as I’m concerned, the main point is that the ObamaCare people, under pressure, agreed to accept paper applications. I downloaded the form, and found that I could print it off with Acrobat 6.04, the most advanced version available for Windows 95/98/ME. That represents eighteen years of backwards compatibility.

The ObamaCare form is not as good as the Internal Revenue Service’s downloadable tax forms, which supports fill-ins with Acrobat 6.xx. When I do my taxes, I start with a text file containing the information for the previous year, make a copy, modify it to reflect this year’s information, paste numbers into the downloaded Acrobat file, and print it off. Most of the information does not change from one year to the next, so this is a quite economical proceeding. From the IRS’s point of view, the advantage of fill-ins is that they are machine-readable, being printed in a designated font and a designated color. As against these technical advantages, the ObamaCare form is good for five years rather than one year.

To my mind, a paper form is the obvious and appropriate way to hand something like taxes, or real estate transactions such as mortgages and leases, or high-stakes insurance. “Put it down in black and white,” they say, in other words, put it down on paper. The web site was an exercise in people playing at being techno-modern in the first place. That was why it proliferated useless JavaScript. The Web works reasonably well for buying small articles on Amazon, with an average value of perhaps ten dollars each. The Web is even better for giving books away, gratis, to the whole world, to anyone who can be prevailed on to read them. You just convert everything into Acrobat or HTML, put it up on your website, create indexes, and you are in business. The Web tends to get into difficulties when you try to use it for things involving serious amounts of money.

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...