Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computers

from the burning-a-hole-in-taxpayers'-pockets dept

The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.

The Commerce Department's Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.

EDA's drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.
Also included in the mass destruction were cameras and TVs. It wasn't just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General's report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.
In an effort to identify infected components, DOC CIRT’s (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information... Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.
Yes. Much like "Reply" and "Reply All" will both get the job done, only one is the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other "indicates" that more than half the EDA's computers are suffering from a malware infection.

DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using the correct network log. Turns out, the original estimate was slightly off.
The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERT’s alert.
This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.
DOC CIRT’s second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.

Specifically, the second incident notification began by stating the information previously provided about the incident was correct. EDA interpreted the statement as confirmation of the first incident notification, when DOC CIRT’s incident handler simply meant to confirm EDA was the agency identified in US-CERT’s alert. Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.

Although the incident notification’s attachment correctly identified only 2 components exhibiting suspicious behavior—not the 146 components that DOC CIRT initially identified—the name of the second incident notification’s attachment exactly matched the first incident notification’s attachment, obscuring the clarification.
For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking at its list (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking at its list (146 components), the EDA responded that reimaging half its devices would be "unfeasible." Taking a look at the EDA's list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.

Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that "over 50%" of the EDA's devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form), both of which were based on DOC CIRT's first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.

The end result? The EDA and DOC CIRT worked together, attempting to head off a "severe" malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the "break stuff" budget had been exhausted.

Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG's findings were brought to the agencies' attention before the fiscal year began and a new "break stuff" budget approved. All in all, the EDA spent over $2.7 million fighting a malware "infection" confined to two computers.

There's nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG's recommendations on agency computer security, with some assessments going back as far as 2006. It's no surprise it managed to (along with the Dept. of Commerce's response team) transform a 2-computer infection into a nearly $3 million catastrophe.



Reader Comments (rss)

(Flattened / Threaded)

  •  
    icon
    Akari Mizunashi (profile), Jul 9th, 2013 @ 6:18am

    I'm betting the equipment was destroyed with $35,000 hammers, supervised by $100,000 consultants.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      Anonymous Howard (profile), Jul 9th, 2013 @ 8:17am

      Re:

      I tend to agree with you on this.

      How the holy fuckin' cow peripherials like mouse and keyboard got destroyed in the process, if not for the sake of "destroying" equipment.

      Probably someone needed stuff and panicked destruction of misreported equipment were the most easiest way.

      1. Look up who the fuck handled the two reports.
      2. Check who did the destroying of the equipment
      3. Check the new equipment vendors

      I bet there would be some connection between 1-2 or 1-3.

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        Anonymous Coward, Jul 9th, 2013 @ 9:11am

        Re: Re:

        Good heavens, I think I knew more about IT when I was 5 years old than these people.

        I actually hope that the equipment was destroyed as part of some backroom deal, and not out of stupidity. If the latter is true, than those responsible should be fired, and banned from ever touching a computer again!

        Seriously, in a case of a malware infection, the most you ever do is format and reinstall, unless the computer in question is junk, and you wanted to get rid of it anyway.

         

        reply to this | link to this | view in chronology ]

        •  
          identicon
          Anonymous Coward, Jul 9th, 2013 @ 9:57am

          Re: Re: Re:

          Well, if it had a backdoor installed it could be hard to close the hole.

          The official response on hijackthis logs is that if you have had certain backdoors, the only way to be certain of avoiding a reinfection is destroying the equipment.

           

          reply to this | link to this | view in chronology ]

          •  
            icon
            Josh in CharlotteNC (profile), Jul 9th, 2013 @ 10:24am

            Re: Re: Re: Re:

            I think you missed this part:

            "Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate"

            Yes, there are some extremely nasty malware variants that are persistant to the (sorta-)hardware level that lodge themselves in the (not-)ROM of a motherboard or other device. But those types were not involved in this incident - it was boring standard malware and email spam. A reimaging of the infected machines would've solved the problem.

             

            reply to this | link to this | view in chronology ]

          •  
            identicon
            Anonymous Coward, Jul 9th, 2013 @ 10:37am

            Re: Re: Re: Re:

            The only way a backdoor can survive a hard-drive wipe is if it is resident in bios, in that case re-flashing the bios should clear it out, the truly paranoid may want to replace the bios chip. There is rarely, if ever, a reason to destroy working equipment.

            By the way, the only way to wipe a hard drive is using old Uinux/Linux tools like dd: dd if = /dev/zero of = /dev/sda, or something similar. The standard windows format command wont do, and can leave malware in hidden portions of the disk (other partitions, boot sector, partition table, etc).

             

            reply to this | link to this | view in chronology ]

            •  
              icon
              John Fenderson (profile), Jul 9th, 2013 @ 10:40am

              Re: Re: Re: Re: Re:

              the only way to wipe a hard drive is using old Uinux/Linux tools like dd


              There are Windows tools to do this as well (there's even a port of dd for those who like to kick it old-school). A half hour in a bulk eraser does a pretty good job, too.

               

              reply to this | link to this | view in chronology ]

              •  
                identicon
                Anonymous Coward, Jul 9th, 2013 @ 10:47am

                Re: Re: Re: Re: Re: Re:

                Of course, but dd is the first thing I thought of, since I use it so often, its such a nifty little program.

                Sorry for the spelling derp, its Unix not Uinux.

                 

                reply to this | link to this | view in chronology ]

          •  
            identicon
            Anonymous Coward, Jul 9th, 2013 @ 12:49pm

            Re: Re: Re: Re:

            or you know, replace the hard drive.

             

            reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 7:52am

    hiding something?

    The kind of paranoia that drives such crazy destruction and money wasting is usually a result of someone who is desperately trying to keep something hidden.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Rikuo (profile), Jul 9th, 2013 @ 7:53am

    I blame out_of_the_blue on this. (S)He's the person who woke up one morning, saw someone had done a search on Google and lost his/her mind (if they even had one to begin with is a question best left for philosophers).

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Trelly (profile), Jul 9th, 2013 @ 7:56am

    Please tell me ...

    ... that the two infected computers were on the list of 146 components that were smashed.

    The only thing that could make the story better is if the original 2 infected computers still remain operational and unmolested by re-imaging.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    TaCktiX (profile), Jul 9th, 2013 @ 8:06am

    The effects of ignorance

    We allow computers to be everpresent while not informing actual users of how to use them in anything but the most basic contexts. If someone who WASN'T CIRT (and preferably in EDA itself) had had a tech-inclined brain at all they would have thrown up a red flag about inconsistencies in this situation and dug up the same thing the IG did without the accompanying waste.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 8:12am

    They should have spent that money on facebook likes instead.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 8:12am

    Uh...where did the money go?

    They blew up $170.000 in equipment. Where's the rest of the money?

    Something smells fishy here. I'm guessing the money went to some contractor that was also conveniently a friend or a family member of someone at the top.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Coogan (profile), Jul 9th, 2013 @ 8:17am

    Is anybody else envisioning a bunch of nerds running around the EDA offices double-time with axes and baseballs bats while the Benny Hill music plays in the background?

     

    reply to this | link to this | view in chronology ]

  • This comment has been flagged by the community. Click here to show it
     
    identicon
    out_of_the_blue, Jul 9th, 2013 @ 8:19am

    I blame economists: "Economic Development Administration"

    The overweening conceit of economists is that they understand everything, and it's all connected, such as Mike held yesterday: that Google being taxed by Germany causes censorship in China!

    http://www.techdirt.com/articles/20130605/06245023322/china-once-again-using-censorship-el sewhere-to-justify-oppressive-great-firewall-china.shtml#c24

    (I see that my point was so effective that Mike hisself made a very rare response to my post. Heh, heh.)

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      Rikuo (profile), Jul 9th, 2013 @ 8:35am

      Re: I blame economists: "Economic Development Administration"

      "that Google being taxed by Germany causes censorship in China!"

      "that Google being taxed by Germany is used as justification by the Chinese government to do censorship over their people"

      FTFY

       

      reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, Jul 9th, 2013 @ 8:38am

      Re: I blame economists: "Economic Development Administration"

      Effective =/= Correct or Truthful at all

       

      reply to this | link to this | view in chronology ]

    •  
      icon
      Rikuo (profile), Jul 9th, 2013 @ 8:44am

      Re: I blame economists: "Economic Development Administration"

      "I see that my point was so effective"

      Basically, you're not looking to say anything truthful or correct, you're merely looking for a response. So, by your criteria, if you were to say that it's all right to rape little kids, and you got responses denouncing you for this bullshit, you would still say that your comment is effective, merely for receiving responses.
      Basically, to you, there is no bad PR.

       

      reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 8:27am

    From the ArsTechnica article on the same story...

    "The EDA's overreaction is, well, a little alarming. Although not entirely to blame—the Department of Commerce's initial communication with EDA grossly overstated the severity of the problem (though corrected its error the following day)—the EDA systematically reacted in the worst possible way. The agency demonstrated serious technical misunderstandings—it shut down its e-mail servers because some of the e-mails on the servers contained malware, even though this posed no risk to the servers themselves—and a general sense of alarmism.

    The malware that was found was common stuff. There were no signs of persistent, novel infections, nor any indications that the perpetrators were nation-states rather than common-or-garden untargeted criminal attacks. The audit does, however, note that the EDA's IT infrastructure was so badly managed and insecure that no attacker would need sophisticated attacks to compromise the agency's systems."

    So this wasn't even an attack. This was on the level of your common variety malware worms and phishing spam stuff that they were destroying HARDWARE over.

    Also note: The NSA in all their infinite wisdom (because afterall THEY should know because they monitor and know EVERYTHING) chimed in to report that they were "concerned" about this. And they should be trusted with keeping everyone's data and using it appropriately?

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 8:42am

    I could of fixed this for free with a little one-two-three punch known as AVG-AdAware-SpyBot.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Rich, Jul 9th, 2013 @ 9:10am

    That's funny. Where I work, in our classified labs, they even put "secret" labels on the mice!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 9:17am

    One Mil

    I would of fixed it for only one million and given them a guarantee too!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    FM Hilton, Jul 9th, 2013 @ 9:45am

    Destruction of brains

    "I actually hope that the equipment was destroyed as part of some backroom deal, and not out of stupidity. If the latter is true, than those responsible should be fired, and banned from ever touching a computer again!"

    If you read the report cited, and the Ars article, it was because the department head did not realize/know/care that it was 'miscommunication' and incorrect information.

    They destroyed nearly half of their systems for not understanding the information, and going insane over it.

    The term "going nuclear" fits this one. They wasted 2.7 million dollars out of paranoid stupidity.

    Those responsible will probably be promoted with a raise. Don't laugh-it's probable.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Jul 9th, 2013 @ 9:52am

    Weird Al's Virus Alert song comes to mind with their reaction.

    http://www.youtube.com/watch?v=zvfD5rnkTws

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    UberSmart, Jul 9th, 2013 @ 10:16am

    A federal budget of trillions and these guys use Windows computers.

    Build your own secret OS and apps, morons.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Berenerd (profile), Jul 9th, 2013 @ 10:22am

    Now I know....

    ...why this song was written...


    http://www.youtube.com/watch?v=zvfD5rnkTws

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    6, Jul 9th, 2013 @ 10:48am

    Well, that is embarrasing.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    aldestrawk (profile), Jul 9th, 2013 @ 2:46pm

    fix root of problem

    The root of the problem is clearly deep-set paranoia brought on by working in a building named after Herbert Hoover. The only true fix will be to tear down this building, build a new one, and never name anything after Herbert Hoover again.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    btrussell (profile), Jul 9th, 2013 @ 3:36pm

    $1.35 million sounds about right. That is what I figure sony owes me for the root-kit they installed on MY computer.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Guardian, Jul 9th, 2013 @ 6:55pm

    cost

    NEW hard drive....80$
    cost to hire a cheap illegal mexican to put it in with a camera watching him 50$

    look on govts face spending 2.7 million
    PRICELESS
    for everything else there is facebook

     

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This