The City Of Baltimore Blew Off A $76,000 Ransomware Demand Only To Find Out A Bunch Of Its Data Had Never Been Backed Up

from the nice-work-if-you-can-get-paid-to-do-it-and-then-not-do-it dept

The City of Baltimore was hit with a ransomware attack in May of this year. Criminals using remodeled and rebranded NSA exploits (EternalBlue) knocked out a "majority" of the city's servers and crippled many of its applications. More details didn't surface until September when the city's government began reshuffling the budget to cover the expenses of recovering from the attack.

The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing.

Johnson, who also serves as the city's chief digital officer, received significant criticism from local authorities for the response to the May 7 attack. City council members alleged a lack of transparency and communication in the wake of the incident, as well as an inability to maintain a functional organization "during an emergency event." He also also never drafted a continuity of operations plan for an IT attack of the kind that occurred.

It looks like the list of stuff Johnson was being paid to do that he never did. Hence the catastrophic outcome when the city refused to pay the $76,000 ransom. Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain.

City residents should be asking WTF their tax dollars are being spent on. The city's audit of its compromised system rolls on, delivering even more embarrassing details about the city's IT skill set. (via Ars Technica)

A new audit of Baltimore’s information technology department says the agency lost key data during May’s ransomware attack because some in the agency used an outdated method for storing files: the hard drives on their individual computers.

[...]

“Performance measures data were saved electronically in responsible personnel’s hard drives,” [Baltimore City Auditor Josh] Pasch reported. “One of the responsible personnel’s hard drive was confiscated and the other responsible personnel’s selected files were removed due to the May 2019 ransomware incident."

Bureaucracies are prone to understatement and the assessment of the ghastly state of affairs by Pasch was no exception. According to Pasch, the permanently-missing data resulted in a "loss of confidence" in the city's IT department's ability to do its job.

This understatement brought the hearing to a halt as council members expressed their disbelief that city data was not being backed up. Their comments were less understated.

Hearing that, City Councilman Eric T. Costello, a former government IT auditor himself, stopped the hearing.

“That can’t be right? That’s real?” Costello asked.

It's apparently real. City data needed for an audit cannot be recovered because the IT department never made an effort to express the dangers of storing the only copy of data locally. It also apparently never made a push to create cloud backups of important files. When the ransomware struck, the stuff locked up was -- in far too many cases -- to only copy of that stuff.

The tragically hilarious postscript to this is the city's response to Ars Technica's request for info on the city's cyberattack recovery plans.

Ars has requested information from the city regarding the contracting details for the recovery, but the city has thus far provided no data. Requests for data on the status of patches and disaster recovery plans were refused because the documents do not exist as a result of the ransomware attack.

It's easy to mock governments for their inability to properly handle the massive amounts of data they collect, create, and retain. And so we shall. The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you're going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec.

Filed Under: backups, baltimore, frank johnson, ransomware


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    That One Guy (profile), 15 Oct 2019 @ 3:42pm

    'Always back up your stuff': A good idea no matter the scale

    I see the idea that everyone needs to learn to back up their stuff the hard way applies to government agencies as well as individuals. Just a pity that people keep having to learn that lesson the hard way, rather than learning from those that came before.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 16 Oct 2019 @ 7:39am

      Re: 'Always back up your stuff': A good idea no matter the scale

      "The Cloud" will let people learn it in a new hard way, when they get banned from a service and find there's no customer support, or get told that a service is shutting down in a few weeks.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 16 Oct 2019 @ 5:26pm

        Re: Re: 'Always back up your stuff': A good idea no matter the s

        "The Cloud" - another name for "somebody else's computer".

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 16 Oct 2019 @ 6:25pm

          Re: Re: Re: 'Always back up your stuff': A good idea no matter t

          "Dude, I'm gonna turn my computer in a couple weeks, ya better get yer shit off by then"
          —various cloud services, paraphrased.

          reply to this | link to this | view in chronology ]

  • icon
    That Anonymous Coward (profile), 15 Oct 2019 @ 3:48pm

    We we're saving money!!
    Backups are just ways to burn money!
    It'll never happen to us.

    reply to this | link to this | view in chronology ]

    • identicon
      A Guy, 15 Oct 2019 @ 3:56pm

      Re:

      If they had a competent IT guy it would have only cost the amount for a few large hard drives and the electricity to download some FOSS backup solutions.

      reply to this | link to this | view in chronology ]

      • identicon
        not frank johnson, 15 Oct 2019 @ 5:24pm

        Re: Re:

        But would a competent IT person have been able to create this linked-in profile?

        https://www.linkedin.com/in/frank-johnson

        Its comically absurd.

        reply to this | link to this | view in chronology ]

        • icon
          R2_v2.0 (profile), 15 Oct 2019 @ 6:32pm

          Re: Re: Re:

          LinkedIn completely checks out:
          Leads all digital transformation programs and supports the Mayor's ongoing efforts to modernize the City of Baltimore's IT capabilities, which also include scaling the local IT ecosystem to drive awareness & tech investment in Baltimore City.

          Now the city has to modernise and invest. Job done.

          reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 16 Oct 2019 @ 12:58pm

          Re: Re: Re:

          Awesome link. Other than all the tech buzzwords, what I see from his profile is that the guy was and still is a salesman. In my experience it is super rare to find a salesman with a firm let alone deep understanding of the tech (in whatever field).

          reply to this | link to this | view in chronology ]

          • identicon
            Anonymous Coward, 16 Oct 2019 @ 5:29pm

            Re: Re: Re: Re:

            Awesome link. Other than all the tech buzzwords, what I see from his profile is that the guy was and still is a salesman.

            Probably be President someday.

            reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 17 Oct 2019 @ 11:48am

        Re: Re:

        It takes far more than a few hard drives and some FOSS software to back up all of the data generated by every employee of every agency for a city the size of Baltimore. The data storage costs alone would be massive, and constantly growing.

        Some other asshole probably thought it should only take very little effort and resources too, so that's what IT got instead of a proper budget and staff, and that's how the city ended up where they are.

        reply to this | link to this | view in chronology ]

        • identicon
          A Guy, 17 Oct 2019 @ 4:20pm

          Re: Re: Re:

          Once again, I said competent IT guy. Most of the information could likely be reduced to text and backed up with a greatly reduced memory footprint.

          Water bills, power bills, typed police/fire/ems reports ect.

          Some information may not be able to have their text extracted and reduced like audio and visual recordings. You only need to save image producing files (i.e. there website formatting or fire report formatting) once also. They don't run youtube or something. The fact is hard drives are huge now and a few of the large ones could likely backup the city of Baltimore.

          You should probably not apply for IT positions.

          reply to this | link to this | view in chronology ]

          • identicon
            A Guy, 17 Oct 2019 @ 7:16pm

            Re: Re: Re: Re:

            Here's 100TB hard drive if you don't believe me.

            https://www.zdnet.com/article/worlds-largest-ssd-hits-100tb/

            reply to this | link to this | view in chronology ]

            • identicon
              Anonymous Coward, 18 Oct 2019 @ 5:23am

              Re: Re: Re: Re: Re:

              You arent in IT obviously, backing up to a 1tb drive, yea thats how it works, 200 bucks all ya need. You have no idea of what a backup structure look like on an Enterprise scale just STFU people like you are why things like this happen.

              reply to this | link to this | view in chronology ]

              • identicon
                A Guy, 18 Oct 2019 @ 5:27am

                Re: Re: Re: Re: Re: Re:

                Apparently you're a hostile idiot. No competent IT person, again, would backup a hard drive image rather than compressible data in that circumstance. You are truly an idiot.

                reply to this | link to this | view in chronology ]

          • identicon
            Anonymous Coward, 18 Oct 2019 @ 12:22pm

            Re: Re: Re: Re:

            I work for a large city the size of baltimore.

            You don't know how hellish it is trying to deal with their data. And it all needs to retain in original formats. Help me

            reply to this | link to this | view in chronology ]

            • identicon
              A Guy, 18 Oct 2019 @ 1:29pm

              Re: Re: Re: Re: Re:

              What is your specific problem? If I was doing a cheap semi-lazy (but competent I hope) job of it I wouldn't bother writing scripts to extract the text and then having the document reconstituted into the proper file structure when the data had to be restored. That would lead to really compressible data but it might be a much larger workload for programmers than you have budget for.

              A cheaper and easier solution that's much lighter on programming is to get a backup solution that downloads required files from pre-specified folders on a schedule (monthly/weekly whatever). Obviously it is a good idea to stagger backup schedules so nothing gets overloaded in the process.

              Finding and running the best compression algorithm for the expected data-type would be my next step. Here is a quick set of replies for the question "what is the best compression algorithm?"

              https://www.quora.com/What-are-data-compression-algorithms?redirected_qid=18888451#

              As some of the answers note there is no one best compression algorithm for every file so some research and experimentation to find good algorithms for the file-types you are backing up may be necessary.

              You would have to get the city departments to cooperate by dropping their work product into the right specified folders and you may have to investigate their computers to find the correct folders to backup too but that can easily be made part of the job. It's not too difficult though it may be time-consuming for a large city.

              FOSS solutions on the market already have the ability to just download a folder instead of imaging an entire drive. You can also make them just download specified file types in a folder if that is more optimal.

              I would also create 2 backup server/computers that run on alternative schedules so if one set of backups is corrupted you have a good chance of recovering from the other, even if is slightly older. You would have a better chance of being able to recover most of the data.

              These are just general thoughts on the issue. If you have a specific problem I would give you my "I hope somewhat competent" opinion if you desire.

              reply to this | link to this | view in chronology ]

  • icon
    Gary (profile), 15 Oct 2019 @ 4:06pm

    Targets...

    The fact that so many municipalities have paid out ransoms recently has painted a huge target on every hospital, town, and county government. They made the right call in not paying the ransom!
    Some insurance companies are just "Insuring" by paying the ransom, causing the problem to spiral downward.

    Now, not backing up. (Or backing up on TAPE), that is the wrong decision.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 15 Oct 2019 @ 4:07pm

    What I don't understand is this: I have backed up my entire home network since the 90s, and use a dual NAS solution plus rotating offsite backups for over a decade. This entire solution cost me a total of around $400 over 10 years ago, plus remembering to take the offsite drive with me when I go to the bank. And I've got redundant backups of all data on all devices on my network.

    Surely an IT department can manage at LEAST this level of data security for their endpoints?

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 15 Oct 2019 @ 4:15pm

      Re:

      Back when I worked in IT, I created a backup solution that merged and deduplicated all files from all endpoints on my networks, and it included a web interface where individuals could use their AD logins to recover any files backed up on endpoints that were under their management. The backup repository could be stored anywhere, as it was encrypted. The software to do so was free; it just took me a few hours to configure and add to local policy for administrative file access.

      Essentially, this will work for any device on a network that allows network access. It also would have been resistant to a ransomware attack, as it was a one-way system. New writes just get deduplicated; deletions required direct access, which means admin account credentials to a locked down system.

      reply to this | link to this | view in chronology ]

      • icon
        Scott Yates (profile), 15 Oct 2019 @ 7:53pm

        What backup system did you use?

        That sounds like a nice solution! I am not aware of one with those features. Would you please share some info on it?

        reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 15 Oct 2019 @ 4:31pm

    The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand.

    Sounds to me that the extortionists need to dramatically increase the amount they demand.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 15 Oct 2019 @ 4:50pm

    I'm sorry, but I must respectfully disagree.

    Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain.

    Some portion of that $6 million (or the estimated $18 million) may be going to data recovery: pulling things off backups, rebuilding data from hardcopy (dead tree data), or going back to original sources and polling institutional memory when no other method works.

    But the rest of it, to "harden" city systems? That's pure technical debt. Money that should have been spent and hadn't been. Systems that are not vulnerable to the latest exploit don't come about by themselves, they get maintained - patched, backed up, put on new hardware when necessary. Security processes get evaluated and updated. Backups get made. Worst case scenarios get gamed out.

    If you wish, consider the 'rebuilding' cost as the cost of a hard security audit. And that cost, as a whole, is likely more expensive than if they'd put a security audit into the budget in the first place.

    reply to this | link to this | view in chronology ]

  • This comment has been flagged by the community. Click here to show it
    identicon
    dmf, 15 Oct 2019 @ 5:16pm

    oh no it's the racisssseses
    they to blame

    dindunuffin

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 15 Oct 2019 @ 7:34pm

    and they thought the IT budget was just fluff

    reply to this | link to this | view in chronology ]

  • identicon
    anonymous coward, 16 Oct 2019 @ 6:43am

    IT for a municipality is VERY difficult.

    I've work as IT for a small municipality (for a short period). The job is very difficult.

    There is minimal budget; trying to get money for backup hardware, storage, time to prepare and test restoration plans is hard (the word impossible is more realistic).

    The situation is worse if one or more public officials doesn't support the effort. Many times municipal IT is merely a political football, being kicked around in the internal political battles.

    All it takes is one official who uses IT as a weapon, who believes that because said official can (barely) do email and create a document, that the said official knows more than IT.

    Part of the villainy is Microsoft. The pretense that Microsoft will take care of it, and maybe some "cloud" backup is all that is necessary has ignorant officials thinking that they are safe at little cost.

    Without the full facts about the performance of the Baltimore's IT boss, Mr. Johnson, we can't really make an informed judgment. It is possible that Frank Johnson didn't do the job he was paid to do. It is also possible that Mr. Johnson did the best he could under difficult and even hostile conditions. In either case the senior officials of Baltimore are to blame. They should have either replaced Mr. Johnson or supported him. Further, the citizens of Baltimore are to blame, they supported the two party status-quo which promise them everything and throw crumbs.

    Further, we are to blame. We sit here complaining. How many of us have fought local, state and the federal government in order to obtain either proper funding for IT, or a mandate to minimize the collection and storage of data. How big an effort have any/all the readers here performed to have a rational data PURGE policy for government?

    So, having been there, and done that and left when I knew that no matter how good or poor I was at IT, I couldn't make city hall understand or accept objective reality. However, at least I tried.

    AC

    reply to this | link to this | view in chronology ]

  • identicon
    Anon, 16 Oct 2019 @ 7:11am

    Business As Usual

    My experience in large organizations is that department heads and other higher level, higher paying jobs are allocated by office politics and favoritism (cronyism?) not competence. IT especially is vulnerable to this, since far more competent people in the field are very poor at skills needed to gain promotions. People tend to understand, let's say, accounting better than IT and the Accounting boss needs a certain level of certification.
    Looks like this guy got a job by being the schmoozer, and then coasted; meanwhile earning brownie points from above by avoiding necessary upgrades in order to demonstrate budget control.
    Like Brownie during Katrina, the relative competence and preparedness is then demonstrated during a crisis that was or was not properly planned for.

    reply to this | link to this | view in chronology ]

    • icon
      lucidrenegade (profile), 16 Oct 2019 @ 9:20am

      Re: Business As Usual

      This is so true. I've working in IT for 25 years and have found that the overwhelming number of failures like this are caused by piss-poor management. From supervisors all the way to the C-level. The IT guys in the trenches get the blame, but they're usually the ones trying to do the right thing while having to fight against said piss-poor management or lack of budget.

      reply to this | link to this | view in chronology ]

      • icon
        lucidrenegade (profile), 16 Oct 2019 @ 9:23am

        Re: Re: Business As Usual

        Not to say there isn't good management in IT, but those folks usually don't last long. They either quit in frustration, or get let go by idiots above them because they didn't toe the party line.

        reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 16 Oct 2019 @ 9:33am

      Re: Business As Usual

      Yeah, I wouldn't be surprised to learn that whoever had the job before Johnson had pushed for redundancy etc., with reasonable price tags, and was pushed out and replaced by someone who could bring the department in under budget.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 16 Oct 2019 @ 7:26am

    Often desktops will have options for data storage, local and some sort of network file. The second is backed up. The thing is, using the network files isn't intuitive to most people's mental workflow, so files go right on the desktop, like they do at home, which may not be backed up. Arguably a training issue, and maybe there could have been better options for document storage as well.

    I'm not sure if this happened here or not but I can also just about guarantee you that the IT department was underfunded and staffed. The question then becomes why?

    The why isn't very tricky, no one sees the network, no one funds the kind of practices and security, and, we / us, whine about overpaid do-nothing government workers. I'm not saying the IT leadership wasn't awful, it can be, and the guy talking about politics has it right, but anytime you see the IT director getting hammered, double-check where the money went, how much of it there is, and make damn sure the city manager and council gets asked hard questions about their involvement in things.

    You might be surprised at what got funded over IT and government is, never, simple.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 16 Oct 2019 @ 7:50am

    From the Ars Technica article; "Johnson, who was hired by the now-resigned Mayor Catherine Pugh, was previously a regional vice president of sales for Intel with no IT operations experience."

    reply to this | link to this | view in chronology ]

  • identicon
    Dave P., 16 Oct 2019 @ 11:58am

    Back-up

    Even on my humble home network of six computers, every PC has several external drives and everything is all on there, duplicated all around the house! Might be a touch of over-kill but that's the way I like it. I do have other anti-ransomware safeguards as well. With over 500 gigs of music, video and docs. on the system, I would not exactly be best pleased if anything happened to that little lot. How come corporations with IT departments and loads-a cash seem unable to engineer reliable back-ups? Is it I.T. versus the bean counters?

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 16 Oct 2019 @ 2:44pm

    “That can’t be right? That’s real?” Costello asked.

    I'm waiting for the punchline, what did Abbott say?

    reply to this | link to this | view in chronology ]

  • icon
    tracker1 (profile), 17 Oct 2019 @ 2:14pm

    Generally poorly staffed govt IT.

    Given the relatively low pay (compared to business/corporate work, usually < 75% of the pay) and the high friction (incredibly lengthy interview/background processes) to get into a government job. It doesn't come close to optimizing for skilled, competent, go-getter types at all.

    The above is why I generally (some military and police exceptions) don't trust people who worked in government for more than a couple years in terms of hiring/interviews. They're horrible systems to work in at times and it's often better to work for adjacent consulting companies in the space than the agencies. Not that it's always the case, some states/counties/cities are better than others.

    In the end, it doesn't surprise me at all. Generally I don't consider something backed up unless it's on 3 different mediums/devices in at least two distinct geographic locations. Beyond this, it's better to drop to a relatively secure location, and have your backup infrastructure pull from that drop location into the backup system. This is a better separation than push, which a compromise like this could effect backups as well.

    Disclaimer, I do work in a company that provides services/hardware/software for govt work. My opinions are my own and do not reflect the company I work for.

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Close

Add A Reply

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Follow Techdirt
Techdirt Gear
Show Now: Takedown
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.