Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computers

Computers

from the burning-a-hole-in-taxpayers'-pockets dept

The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.

The Commerce Department’s Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.

EDA’s drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.

Also included in the mass destruction were cameras and TVs. It wasn’t just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General’s report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.

In an effort to identify infected components, DOC CIRT’s (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information… Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.

Yes. Much like “Reply” and “Reply All” will both get the job done, only one is the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other “indicates” that more than half the EDA’s computers are suffering from a malware infection.

DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using the correct network log. Turns out, the original estimate was slightly off.

The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERT’s alert.

This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.

DOC CIRT’s second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.

Specifically, the second incident notification began by stating the information previously provided about the incident was correct. EDA interpreted the statement as confirmation of the first incident notification, when DOC CIRT’s incident handler simply meant to confirm EDA was the agency identified in US-CERT’s alert. Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.

Although the incident notification’s attachment correctly identified only 2 components exhibiting suspicious behavior—not the 146 components that DOC CIRT initially identified—the name of the second incident notification’s attachment exactly matched the first incident notification’s attachment, obscuring the clarification.

For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking at its list (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking at its list (146 components), the EDA responded that reimaging half its devices would be “unfeasible.” Taking a look at the EDA’s list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.

Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that “over 50%” of the EDA’s devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form), both of which were based on DOC CIRT’s first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.

The end result? The EDA and DOC CIRT worked together, attempting to head off a “severe” malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither “persistent” nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the “break stuff” budget had been exhausted.

Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG’s findings were brought to the agencies’ attention before the fiscal year began and a new “break stuff” budget approved. All in all, the EDA spent over $2.7 million fighting a malware “infection” confined to two computers.

There’s nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG’s recommendations on agency computer security, with some assessments going back as far as 2006. It’s no surprise it managed to (along with the Dept. of Commerce’s response team) transform a 2-computer infection into a nearly $3 million catastrophe.

Filed Under: commerce department, malware, miscommunication