IRS Still Working To Complete Computer Upgrade From Windows XP To Windows 7

from the by-2020-it-will-be-2010 dept

Like many other entities (both singular and conglomerate), the IRS was reluctant to sever ties with Windows XP. Microsoft forced the issue, however, and gave everyone plenty of time to migrate to an operating system released sometime in the last ten years. Even with this head start, the IRS has yet to meet this target.

An Inspector General’s report notes that the IRS is almost finishing upgrading its workstations to an operating system that’s only eight years old (Windows 7). Almost.

As of May 2015, the IRS has completed most of the Windows XP workstation upgrades across the country. Approximately 1,300 workstations have yet to be located or confirmed as running the old operating system.

At this point, I’m going to do something I rarely do: cut a government agency some slack. The IRS did have plenty of workstations to upgrade — nearly 110,000 — so if 1,300 went “missing,” it’s somewhat understandable. On top of this, budget issues forced the agency to upgrade old workstations instead of replacing them with newer systems, which would have greatly sped up the process.

The IRS claims it does know where these missing 1,300 workstations are, but that the Inspector General won’t listen to it. The included “Management Response” says the following:

The audit incorrectly concludes that IRS has not accounted for all XP workstations. We acknowledge there were challenges with our inventory data due to the many antiquated systems in our IT ecosystem. In spite of this, we took extraordinary steps to identify, document and upgrade every XP workstation in the IRS. On several occasions throughout the audit, the IRS provided information to the TIGTA team that clearly documented the number of workstations to be upgraded, where those workstations were located, and our strategy to complete the upgrades. Although footnoted in the report, TIGTA opted not to change their assertion that the IRS had not accounted for all XP workstations. As of this date, only 71 Windows XP workstations remain to be migrated.’

The IG’s footnote tells a different story.

After the conclusion of our fieldwork, the IRS provided documentation that these workstations were located and upgraded to Windows 7, as of July 22, 2015. We were unable to verify this information.

Beyond the workstations, there’s the IRS’s servers, which are also running up against Microsoft’s upgrade clock. This not-overly-optimistic statement by the IG suggests the IRS will be living in the (OS) past for much of the future.

Based on our discussions with management, we determined it is unlikely that the IRS will have its servers upgraded to Windows Server 2012 any time this Fiscal Year.

This is due to the fact that the IRS is still struggling to upgrade its servers to seven-year-old software.

In fact, the IRS still has not fully upgraded its servers from Windows Server 2003 to the 2008 release. Currently, the IRS has approximately 3,000 Windows servers still running the 2003 operating system. Management informed us that they have upgraded approximately 4,100 servers to the 2008 version which is already seven years old. The IRS currently has no servers running the 2012 operating system in production at this time.

Time to start reeling in the slack I cut the agency earlier. This logistical issue seems especially absurd.

The IRS also discovered nearly 6,000 applications being used by employees to do their jobs that required an assessment of each application to determine whether it would operate on Windows 7.

Unfortunately, the report doesn’t provide more details on the massive amount of applications being used by the IRS. Every interlocking piece presents a new possibility for a hole or an exploitable flaw, something compounded by the use of unsupported system software.

The IRS has already seen its system exploited by scam artists, who were able to use the credentials of taxpapyers to fraudulently obtain refunds. That its “user data” (the tax records and personally-identifiable information of millions of Americans) is secured behind a patchwork of outdated software presents criminals and rival governments other opportunities for exfiltration and exploitation of taxpayer data.

Even if the IRS manages to hit its self-imposed targets for the most recent round of upgrades, support for those operating systems is also on its way out.

Despite the eventual progress made by the IRS on the Windows XP upgrade efforts, we believe the IRS provided inadequate oversight and monitoring during the early phases of this effort, starting with including it among other Microsoft product upgrades rather than making this effort its own project up to the decision made by the CTO to oversee the project himself. In addition, after taking four years to upgrade to Windows 7, the IRS is now faced with the challenge of addressing Microsoft’s announcement to end extended support for Windows 7 in January 2020.

The IRS has agreed to a majority of the Inspector General’s recommendations which means… well, it probably doesn’t mean much of anything. Chances are the IG will revisit this in a few years and still see the agency struggling to stay current with its operating system software. It’s eight years behind on system software and seven years behind on server software, with the latter’s migration less than 50% complete. The IRS doesn’t have it easy, not with 110,000 workstations, 7,000 servers and — for god knows what reason — 6,000 applications, but unless it’s willing to give this the priority it deserves, it will always be in danger of making a flawed, bulky system even more insecure.

Filed Under: , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “IRS Still Working To Complete Computer Upgrade From Windows XP To Windows 7”

Subscribe: RSS Leave a comment
Anonymous Coward too says:

Re: As we learned this week

Microsoft might just do it for them.

Ahem. I think that brilliant comment is lost on most readers.

If you keep up with the tech news you will know that Microsoft automatically fetching Win 10 and caching it so that Win7/8 can update. In the past week it’s come out that Microsoft will force those updates on everyone.

Rekrul says:

Re: Re: As we learned this week

If you keep up with the tech news you will know that Microsoft automatically fetching Win 10 and caching it so that Win7/8 can update. In the past week it’s come out that Microsoft will force those updates on everyone.

And then once all the systems are upgraded to Win10, they can start sending potentially confidential taxpayer information from those systems back to Microsoft.

The Root/Rootbrian (user link) says:

Why not go GNU?

Seemingly being the first to comment and all, and being an active GNU/Linux user of the Mint distribution, I would expect the IRS to keep using the same hardware, only to replace parts when they physically wear out/fail. Replacing entire systems is a big waste of money.

I used the same hardware for a total of 13 years until something affected the board on mine, and it refused to turn on (after everything was tried), then I got a used one and have still been using it. No problems.

Maybe consider moving away from microsoft’s monopoly and choosing free software would be a better thing, and last but not least, far less exploits due to constant patching. Imagine how much money it would save (no licenses, activations or product keys!), and since updates can easily be configured to be unattended (I myself haven’t made use of that, too advanced for casual users!), it can happen when the offices are closed.

I would hate to see the servers running windows too, when ubuntu server (or another server client, be it debian server?) be installed and then maintenance can go through much easier. Then upgrading can be fool-proof, easily done. I wish they would consider this.

Anonymous Coward says:

Re: Why not go GNU?

There is no question whatsoever that upgrading to open source software was and is the correct move. The debate is only over which operating system platform would be best-suited. Having using them all in a variety of environments over the past several decades, I suspect that a mix might work best: BSD on the servers, Linux on the desktops. But ascertaining whether or not that guess is accurate would require far more detailed analysis, of course.

The most obvious benefits of such a move are security and cost. Closed-source operating systems are insecure by design and cannot be fixed; open-source operating systems are not necessarily secure, but they provide a fighting chance. Closed-source operating systems are extremely expensive to maintain, especially at scale (witness this article); open-source operating systems are vastly cheaper both to run and to upgrade.

But there are other benefits as well: open-source operating systems run well on older hardware (I’m typing this message on a Lenovo laptop that’s 8 years old) (and it’s not my oldest one) and make optimal use of hardware resources. They are unencumbered by the spyware that is now not only epidemic in applications, but part of Windows 10. They enjoy incredible, long-lived support and there is an enormous pool of talent out there skilled at debugging them. They also make a serious effort to comply with standards, whether those are protocols, file formats, or anything else — thus they’re highly interoperable.

The IRS should have left Windows in its read-view mirror over a decade ago. Every dollar spent on this “upgrade” is wasted.

Gwiz (profile) says:

Re: Re: Re: Why not go GNU?

It will be met with opposition though.

And not just from the IT guys. From the accountants and managers too. I work at a small shop for a guy who used to be an accountant. I’ve attempted to persuade him that going the GNU route on our workstations would be beneficial in the long run, but I’m always met with the attitude of “if it doesn’t cost a lot of money then it cannot be of any value” from him. It’s a pretty difficult mindset to combat sometimes.

Anonymous Coward says:

Re: Re: Re:3 Why not go GNU?

I’ve often heard the “who do I sue?” question.

When I point out that — so far — no customer who has been given clearly-defective software by Microsoft or Oracle or Apple or IBM or or or or has successfully litigated against them to recover damages, that question tends to go away.

Socrates says:

Re: Re: Re:4 Why not go GNU?


and people get in the same trap again nowadays!

“Put your files in the cloud”. Are people insane, even several 100.000.000 user companies like Megaupload is unable to protect the files.

“Sync your devises”. Are people even more insane. Something might delete things on all your devices automatically.

Socrates says:

Re: Re: Re:2 Mindsets

To me religion is more than assassinating a goat to be blameless for various crimes, or drinking a cup of wine and pretending that it is the blood of Jesus. It is also cultural belief-sets in a wider sense.

If an accountant balance books all day long for years, it would be natural for him to believe that value on one side is balanced with some other value on the other side, in the real world too. As even goodwill may be post in the sheets, why shouldn’t he believe so?

If a manager want something done, he have to fund it. If he provide to little funding it takes longer and may cost more in the long run. And the return on investment will be delayed too. Why shouldn’t he believe that there must be a close relation between cost and value?

And most people experience that if something is too good to be true, it usually is.

This is the most dominating belief-system in our western world.

Fallacies play a vital role in most belief-systems, this one including. It fails to consider that there is a lot of value we doesn’t pay for, friendship, close relationships, the air we breathe. We may take it for granted as we is so used to only value things with price tags. Until we loose something, then it might be to late.

It is just too easy to focus only on subjects that has to be fixed, because they demand effort to be moved along, and forgetting to appreciate what we have.

Sometimes we can choose to pay or get something for free, be it sex, software, and many other things. Things that is too good to be true, might be the best things in life!

tqk (profile) says:

Re: Re: Why not go GNU?

The IRS should have left Windows in its read-view mirror over a decade ago.

Yes, they should’ve, however given their masters’ (Congress) vulnerability to (or dependence on) corporate lobbyists, can anyone really be surprised it isn’t an option? Oracle Corp. alone could get them all crucified easily with a few FUD doom and gloom press releases. Corporate IT is rabidly pro-proprietary. It took massive amounts of begging for years just to convince them to try it on servers. That, and a lot of do it and don’t tell ’em until you can prove to them it works.

Add to this biting the bullet and migrating to a new system like this is a huge leap for these people. They’re convinced that (eg.) migrating from MS-Office to LibreOffice would mean completely throwing away whole skillsets requiring complete (and horrifically expensive) retraining for users. The cost of retraining users to go from one version of Windows to another is already massive. I’ve never understood why users shouldn’t be expected to retrain themselves, but I’ve never worked in HR or management.

It’s too bad none of them can even consider getting ahold of Munich Germany’s Linux distro and running a pilot program through the many iterations it would take to get it into use. This isn’t the way managers like to do things though. They want to be in the herd, not leading it.

Cody Jackson (profile) says:

Re: Why not go GNU?

There are multiple reasons, but I think the primary reason the government doesn’t switch is because of inertia. You already know the gov. has a hard time switching gears; imagine trying to retrain thousands of people on a new operating system.

Granted, every Windows update has a new GUI for users to figure out, so IMO making the switch to Mac or Linux isn’t any different. However, the applications will have to change as well. There are *nix compatible version of popular Windows software, but “their just not the same as Office”, even though people only use a small number of the features.

People just don’t like change, and the people in a position to make the change are the least likely to want it, especially if it affects them. I can’t tell you how many policies I’ve seen that are immediately ignored by those in power. They may pay lip-service to it, but in practice they don’t follow it, or find a way to get around it.

Another significant factor is Exchange email servers. These agencies have invested significant resources in an Exchange infrastructure. Currently, I’m not aware of any open-source equivalent to Exchange that can be a drop-in replacement. Since email is the main work-tool for most people (and frequently a storage medium as well), not being able to replace Exchange is a deal-killer.

Finally, there just aren’t that many people trained in *nix, and those who are get well-paid. As you may have heard, the gov. is not a place to get wealthy, so the people with the necessary skills will go where the money is, i.e. the private sector.

At the place I currently work, they haven’t had a bona-fide developer there for at least five years, but probably longer. They have temporary workers, who might stay for a couple of years before moving on. Management isn’t willing to pay a programmer what they are worth to maintain all the systems, so they have to make do with band-aids and duct tape.

I’m sure the IRS is similar: get the most work with the least money. Switching to Linux, while intelligent for many reasons (and may have been advocated at times), simply won’t happen because no one with authority is interested in it. If something bad happens, they will be in the spotlight, so it’s easier to make excuses than fix the underlying problem.

Skeeter says:

Re: Why not go GNU?

GO LINUX OR GO AMATEUR! This was the first thought I had, when reading this Microjunk nightmare! Why is the government constantly on the dole to throw billions at a private company all the time? Why not hire the programmers, start with a base install of some flavor of Linux (maybe an Ubuntu generic desktop for the push-button lackies), and build their servers on the UNIX/Linux platform that they not only WOULD OWN, but could audit, edit, upgrade, roll-back (whatever), whenever they wanted to?
I mean, of all things, this is an absolute NO BRAINER that would give them far more security, more stability, and forever burn that demonic ‘Microsoft End-Of-Support’ calendar that everyone gets beaten to death by.

All government systems should be Linux-based with in-house programming. Anything else is a tremendous security risk, not to mention a massive monetary-black-hole.

Anonymous Hero says:

I agree!

> At this point, I’m going to do something I rarely do: cut a government agency some slack.

I agree! This is no easy task, because first you have to destroy all the hard drives, and the you have to…

(kidding aside, there is the unfortunate truth that not only do they have to upgrade 110,000 machines, but they have to do so at the speed of bureaucracy)

Rekrul says:

Re: I agree!

I agree! This is no easy task, because first you have to destroy all the hard drives, and the you have to…

Yup! Everyone knows that it’s impossible to ever erase anything off a hard drive. If you know what you’re doing, you can retrieve every bit of information that’s ever been saved to that drive, no matter how many times it’s been overwritten! CSI said so!

Skeeter says:

Re: Re: Re:

Actually, in reading this IG report, I wonder how much of that got left off the report, being as technically, some systems running WinX would be ‘upgraded’ past 7.

It doesn’t matter that Win X is a MAJOR security risk, and that all I’ve talked to with it deployed have only kept it there because they were ‘made to’ by administrative management that ‘just don’t get it’.

I bailed on supporting Windows in any flavor after they tried to axe Win 7 after only 3-years in the market. I saw it coming to a ‘rolling 18-month product window, eventually’ and said ‘yep, stop the psycho train, I want off here’. Moving my whole network and related support systems over to Linux as the best pain I’ve had in a long time, and I estimate about $22,000 saved in licensing fees alone, not to even bring-up the ‘you have it, you own it’ security of knowing I don’t need to call anyone to renew keys (and argue with them about them) if I have to reinstall’.

Microsoft took the idea of a ‘purchased intellectual property’, tried to treat it like a sports car you would buy, then shell-gamed the world into turning a purchase contract into a rental agreement with a LOT of rights-lost loopholes that now impinge on the 4th and 5th Amendments. That they are up to no good is clearly seen in them wanting to now include ‘Linux Bash’ (the Linux Terminal) into their Windows product. WHY?! Microsoft has ‘Terminal’ that they took all your DOS command line abilities away from. Ask yourself why they would now want to allow you to reach into a Linux Terminal, even to be able to write code, when the don’t even want you to have the ability to create an MSDOS boot USB stick from their own Terminal? Then only answers to this are very nefarious, indeed.

Socrates says:

Big complex systems

Migrating big complex systems give ample opportunity for Murphy to show that “what can go wrong…”. It might feel safer to stay as close as possible to IE6, XP, MS Office, and software that depend on them. Minimizing change lessen temporary productivity drop and costs relating to education.

But, sadly, the new system will inherit more than familiarity; it will inherit the problems too.

IRS should seriously consider to move towards Linux/FreeBSD/OpenBSD on the servers first, and then start the work of moving the users over too. It should consider how it stores information in a way that still works decades and several migrations later. It is important to take extra care of the employees during any change, but “standing in the Microsoft cement, waiting for it to harden” is a recipe for needless suffering.

Anonymous Coward says:

Re: Big complex systems

Changing operating system vendors doesn’t fix the problem, that the IT department is poorly managed. 110000 systems sounds like a lot, but other government agencies and large companies have managed to do this.

Imagine if they just now finished upgrading systems to Ubuntu 9.10, it would be just as bad, if not worse, than being so out of date with their Windows Version.

The real problem is that they think that they have completed something. Keeping current is something that you always do. Windows 7 is already two generations old, if they are not already working on their Windows 10 deployments, they are already behind, for no other reason than they haven’t started yet.

Socrates says:

Re: Re: Big complex systems

Changing operating system vendors doesn’t fix the problem

Migrating to systems that honor standards, is not “Changing operating systems vendors”. It is actively choosing solutions that does not rely on any one vendor.

GNU software tend to work across versions, across OS “flavors”, with different servers, protocols, and so on. Partly this is a philosophical choice, based on how the developers want it to work. But it is also by necessity, the developers (and users) is spread among the “flavors” and the software have to respect common standards.

This translates into a massive benefit for the users (and for sys-ops).

Elimination of license administration, tools such as apt-get, and ability to update without incurring costs makes staying current much easier.

Skeeter says:

Re: Big complex systems

AMEN! Linux is backwards-compatible for at least the last 10-years (some of it is backwards compatible all the way back to ‘Great Granddad’, AT&T’s last version of UNIX 7 (technically, Bell Labs)).

If you have Linux installed, and you hire a programmer, you have TOTAL AND ABSOLUTE CONTROL over your system, FOREVERMORE! Linux doesn’t ‘expire’, it is ‘free’, it has a ‘desktop’ (for users), and it is VERY flexible (not to mention far more virus-resistant, more friendly to your hardware, etc.)

I know what the problem really is, they are now in bed with Microsoft, and if they try to leave, they worry about what Microsoft knows that it would possibly ‘leak’ or ‘disclose’ as a result of that much loss-of-income. Or, maybe it’s just that word, ‘FREE’, that the government is choking on…I’m not sure which.

Anonymous Coward says:

I do see why you care that wore no quick to remove XP or that they have chosen Windows 7 as the replacement. The US government should be in no rush to move Windows 8 or higher, when those operating system are laden with iOS-like DRMcensorship and as your-self as discussed in other articles, spyware. Windows 7 is a bit better if take not to install the “telemetry” updates (or install DWS) but, it would be nicer to see them stick with XP or try their hand with something like PC-BSD or some from of Linux such Descent|OS.

Anonymous Coward says:

No idea what it’s like at the IRS, but I can speak from experience in a different 3-letter government agency in the early 2000s. Even then, most of their field offices were using computers for which there was no mouse or proper GUI, basically just one step removed mainframe terminals. I worked as a consultant on a project to modernize those systems, shortly before the whole thing was axed due to mismanagement and running way over budget with nothing to show for it. What do you expect when the only sizable contractors with the necessary clearances are all ex-military, with no private sector experience? Certainly not “industry best practices”.

Not one of the engineers I encountered would be able to keep an equivalent job in the real world. In this kind of government work, the development cycles are painfully slow and unresponsive, everything is overengineered with no short-term usability goals (or attainable goals in general), internal communication is almost nonexistent, wheels are constantly reinvented, and they don’t use any off-the-shelf development or productivity tool unless it was from an approved vendor and was procured 3+ years prior with a 15-year support contract. The layers of bureaucracy for any systems they manage themselves are not designed to accommodate keeping anything updated or patching security holes like you are used to in the real world.

Those of you saying why don’t they just use GNU, BSD, whatever…again, they have no private-sector experience, so many of those older military guys just think those are toy/hobby systems for kids and black-hats. If they get over that, then they rail against the licenses and the idea of code being open source. If they get over that, then they say they’re too committed to existing technologies, they can’t switch mid-stream now, yada yada.

To their credit, 4+ years into the next version of the project, when it became apparent they were repeating the same mistakes, they finally dumped the mega-contractors and cranked out their own system in-house with agile processes, off-the-shelf tools, and modern computers. It finally rolled out in 2012. Maybe the IRS will learn from them.

Skeeter says:

Re: Re:

I have trouble with some of your suppositions in what you say. “so many of those older military guys just think those are toy/hobby systems for kids and black-hats.” – well, if I told them that Linux was the new generation UNIX, think that would change their perspective? I mean, in reality, that’s exactly what it is.

No one in their right mind could ever say ‘UNIX’ can’t get the job done, so why do you think ‘Linux’ can’t? The argument doesn’t even get off the launch pad. So, if I just justified to you the ‘seriousness’ of Linux in one small response, why doesn’t the government buy into it? THIS is the question that you have to take a deep look into, and it is not about ‘do-ability’, it’s about graft, multi-level profiteering, and corruption. Of course, they will always try to throw the shoe one the other foot and say, ‘our workforce is too technically illiterate to work in Linux then’, but being as Apple iOS is in many ways, another flavor of UNIX too, that argument doesn’t fly either. (seriously, I know people that are Apple ‘button-pushers’, too, and trust me, ignorance can become proficient at ANYTHING that benefits them).

In the end, it’s the same old story – follow the money, watch out for those deep holes of corruption and spying you might step in along the way.

I’ve personally always thought that the number one reason that UNIX/Linux isn’t preferred (other than corruption/money) is that it is far more secure than Windows, and you can’t justify spying in a system that you can lock down that securely, and trojans are near-impossible to install ‘accidentally’ in it.

I hate Microsoft says:

Microsoft Should Be Dismantled

I believe the IRS should be looking into Microsoft’s attempt to evade taxes. Microsoft has no sovereignty to force these American Government Agencies that are supposed to be serving the American public into a wasteful mode costing the American taxpayers billions of dollars just so it can increase its profits, totally dissing America and its government.

Skeeter says:

Re: Re:

Actually, if you replace ‘Server 2016’ with Linux 15.10, and then recommended a new ‘Federal Redevelopment Program’ to make a central IT resource (akin to the ‘New Deal’ mindset) in, say, Dallas; and then dedicate to NOTHING but programming new systems for Linux Platform ONLY, for Government Systems – THEN you would have my absolute buy-in.

You would create jobs, create new software, break the monopoly (and related power-control-center that a few mega-corporations have on government), and at the same time, be able to have an editable, auditable software system that our nation would OWN, could MODIFY, and could KEEP as they needed. It is a win-win this way.

Anything else is paying blackmail to private profiteers to ‘not turn our lights off yet’, while racing against that wet-concrete clock of ‘bureaucracy’.

Klaus says:

6000 applications?

“The IRS also discovered nearly 6,000 applications being used by employees to do their jobs that required an assessment of each application to determine whether it would operate on Windows 7.”

This depends on how they’ve defined “application”. It may for example include user developed Windows batch-files that have become essential to their day-to-day running, in which case 6000 could be plausible.

Skeeter says:

6000 applications?

Klaus, I agree with what you imply.

There is NO WAY they have ‘6000-applications’ that they are individually using, unless they are counting each window as it opens.

In all likelihood, they are counting TONS of script-kiddie / command-line batch files that are nothing more than opening one window and closing another. The real ‘cost-applications’ would impact maybe 25-50 actual commercial programs, the rest are either ‘in-house’ scripts or ‘band-aids’ meant to bypass problems from when they created this IRS computer beast 20-years ago based on Microsoft technology (where they should have stayed with UNIX/Linux and developed their tools in-house anyhow).

Bureaucracy just can’t learn the lesson that it cannot move faster than private profiteers. Then again, if you made them pay for it out of their own pockets, they would fix it tomorrow, too.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...