The Story Behind The XZ Backdoor Is Way More Fascinating Than It Should Be

(Mis)Uses of Technology

from the the-long-con,-the-short-reveal dept

Every few years, it seems, we’re reminded of the incredible number of dependencies built into the software we all rely on. Remember kik? Or Chef Sugar? Or any number of similar situations? The xkcd comic on dependency is so well known for a reason.

So, the backdoor that was discovered in xz utils a week and a half ago felt somewhat familiar in some ways. It was another case where there was a dependency that relied on basically one random dude keeping some code up to date.

But, in many important ways it was really different. I took a bit of time before writing about this, because so much time was spent last week by people trying to sort out the details, and I was a little afraid that it was too early to understand what had happened. However, by now it’s looking pretty clear.

xz Utlis is a data compression tool that’s found in nearly all versions of Linux. On Friday, March 29, Andres Freund sent an email to a security mailing list, laying out why he believed there was a backdoor built into it. In an interview with the NY Times, Freund reveals just how close we were to no one finding this:

The saga began earlier this year, when Mr. Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed a few error messages he didn’t recognize. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory.

But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to log into computers remotely, was using more processing power than normal. He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to the earlier errors he’d seen.

So, because it was using slightly more processing power than normal, he started digging deeper.

In particular, he found that someone had planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.

And even then, he wasn’t really sure and hesitated:

At first, Mr. Freund doubted his own findings. Had he really discovered a backdoor in one of the world’s most heavily scrutinized open-source programs?

“It felt surreal,” he said. “There were moments where I was like, I must have just had a bad night of sleep and had some fever dreams.”

And that’s where the mystery got even deeper. Like lots of open source software, lots of people suggested changes, and starting in 2021 a user with the username “JiaT75,” known as Jia Tan, had been submitting code to various open source projects. In early 2023, they submitted to xz Utils.

Soon after that, what appears to be a sneaky social engineering scheme began. As described by Dan Goodin at Ars Technica, a bunch of random people started complaining that the long-term maintainer of xz was falling down on the job. This led others to suggest that perhaps they needed help and pointed to Jia Tan as a possible helper.

It would appear that this backdoor was years in the making. In 2021, someone with the username JiaT75 made their first known commit to an open source project. In retrospect, the change to the libarchive project is suspicious, because it replaced the safe_fprint funcion with a variant that has long been recognized as less secure. No one noticed at the time.

The following year, JiaT75 submitted a patch over the xz Utils mailing list, and, almost immediately, a never-before-seen participant named Jigar Kumar joined the discussion and argued that Lasse Collin, the longtime maintainer of xz Utils, hadn’t been updating the software often or fast enough. Kumar, with the support of Dennis Ens and several other people who had never had a presence on the list, pressured Collin to bring on an additional developer to maintain the project.

In January 2023, JiaT75 made their first commit to xz Utils. In the months following, JiaT75, who used the name Jia Tan, became increasingly involved in xz Utils affairs. For instance, Tan replaced Collins’ contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited. Tan also requested that oss-fuzz disable the ifunc function during testing, a change that prevented it from detecting the malicious changes Tan would soon make to xz Utils.

As Wired notes, this appears to be a slow burn operation, likely state sponsored, using the openness of open source technology, combined with the social engineering to slip in this very dangerous backdoor.

That inhumanly patient approach, along with the technical features and sophistication of the backdoor itself, has led many in the cybersecurity world to believe that Jia Tan must, in fact, be a handle operated by state-sponsored hackers—and very good ones. “This multiyear operation was very cunning, and the implanted backdoor is incredibly deceptive,” says Costin Raiu, who until last year served as the most senior researcher and head of the global research and analysis team at Russian cybersecurity firm Kaspersky. “I’d say this is a nation-state-backed group, one with long-term goals in mind that affords to invest into multiyear infiltration of open source projects.”

As for which nation, Raiu names the usual suspects: China, Russia, and North Korea. He says it’s still too early to know the true culprit. “One thing is for sure clear,” he adds. “This was more cunning than all previous software supply chain attacks I’ve seen.”

I’ve seen some suggest that this is why open source software is vulnerable, since this sort of attack can happen (and there are already some stories suggesting at least attempts elsewhere). But, of course, the fact that it’s open source software is also why the backdoor was discovered (and relatively quickly).

So, as it stands, it appears that a long con scam was under way. This scam allowed someone (or some agency) to get extra power over some random dependency found in most versions of Linux, through social engineering. That involved both submitting useful code to the utility over time, and then having a group of users complain that the existing maintainer was being too slow to fix things, and suggesting this one contributor who had been useful and had a multi-year track record.

Then, as that user gained more and more trust — and control —, they were eventually able to slip in a backdoor that had the potential to be massively dangerous. It was only stopped because one dude found that some process appeared to be running a bit slow.

As security expert Alex Stamos told the NY Times:

If it had gone undetected, Mr. Stamos said, the backdoor would have “given its creators a master key to any of the hundreds of millions of computers around the world that run SSH.” That key could have allowed them to steal private information, plant crippling malware, or cause major disruptions to infrastructure — all without being caught.

And, of course, it’s entirely possible that similar attacks have already been successful. However, after what was discovered by Freund over the last few weeks, and the collective hand-wringing in the open source and computer security worlds, it’s hopefully much more difficult for it to occur as people are now much more aware of this possible attack.

This isn’t a story of someone getting fed up or annoyed at how others were using their software, unlike those past stories about dependencies. Hopefully, we never have to find out just how disastrous this could have been by having a similar attack succeed (or at least not get caught this quickly).

Filed Under: backdoors, jia tan, open source, xz utils