3CX Knew Its App Was Being Flagged By AV Platforms, Did Very Little During Supply Chain Attack
from the whoops dept
If you don’t use the 3CX VoIP platform, or work in the MSP space with companies that do, you may have missed the news that the company suffered a massive supply chain attack over the past few days. With comparisons being made to the SolarWinds fiasco, this was really, really bad. Unsuspecting clients of 3CX had Windows and Mac versions of the app to hundreds of thousands of customers deployed on their computers with malware snuck inside. That malware called out to actor-controlled servers, which then deployed more malware designed to allow for everything from browser hijacking to remote-takeover of the computer entirely. A hacking group associated with the North Korean government is suspected to be behind all of this.
Security firm CrowdStrike said the infrastructure and an encryption key used in the attack match those seen in a March 7 campaign carried out by Labyrinth Chollima, the tracking name for a threat actor aligned with the North Korean government.
The attack came to light late on Wednesday, when products from various security companies began detecting malicious activity coming from legitimately signed binaries for 3CX desktop apps. Preparations for the sophisticated operation began no later than February 2022, when the threat actor registered a sprawling set of domains used to communicate with infected devices. By March 22, security firm Sentinel One saw a spike in behavioral detections of the 3CXDesktopApp. That same day, 3CX users started online threads discussing what they believed were potential false-positive detections of 3CXDesktopApp by their endpoint security apps.
Here’s the problem with that last paragraph: the detections for the malicious code actually began before Wednesday, March 29th. In an updated ArsTechnica post, it turns out that customers were noting that some AV agents were flagging the 3CX installer and app going all the way back to March 22nd, a week earlier. And these customers were noting this on 3CX’s own community forums.
“Is anyone else seeing this issue with other A/V vendors?” one company customer asked on March 22, in a post titled “Threat alerts from SentinelOne for desktop update initiated from desktop client.” The customer was referring to an endpoint malware detection product from security firm SentinelOne. Included in the post were some of SentinelOne’s suspicions: the detection of shellcode, code injection to other process memory space, and other trademarks of software exploitation.
Others were, in fact, seeing the same thing. These customers were busy writing exceptions for the application, figuring that a signed/trusted app from the manufacturer itself was likely resulting in a false negative. Other users followed suit. 3CX remained silent until Tuesday, March 28th.
A few minutes later, a member of the 3CX support team joined in the discussion for the first time, recommending that customers contact SentinelOne since it was that company’s software triggering the warning. Another customer pushed back in response, writing:
Hmmm… the more people using both 3CX and SentinelOne get the same problem. Wouldn’t it be nice if you from 3CX would contact SentinelOne and figure out if this is a false positive or not? – From provider to provider – so at the end, you and the community would know if it is still save and sound?
This is, of course, precisely what should have happened. Instead, the 3CX rep said there were too many AV providers to go out there and call them all. Then he or she mentioned that they don’t control the antivirus software, but instructed the user to “feel free to post your findings” once they had called SentinelOne themselves.
Those findings were on display for everyone the following day when the attack and compromise of 3CX became very, very public.
You really would think that after SolarWinds first and Kaseya second, tech companies would know better than to ignore this sort of thing and actually talk to the security firms that are flagging their products.
Filed Under: antivirus, hack, supply chain attack, vulnerability
Companies: 3cx
Comments on “3CX Knew Its App Was Being Flagged By AV Platforms, Did Very Little During Supply Chain Attack”
Well on the upside, they no longer have any clients to worry about pissing off.
Re:
“This job would be great if it wasn’t for the fucking customers.”
Do you have any actual information, though, or are you just being optimistic? I’m sure there are hundreds of companies that have had terrible security flaws, botched their handling, and are still in business. If it’s especially bad, they might change their name.
Re: Re:
I have little hope that people will do anything correctly.
What I can tell is that if North Korea owned your entire network because 1 company screwed up, told customers to look into the issue themselves b/c there are to many AV companies, and report back the findings to them I think this would be a big reasons people might have to close their companies & never use 3cx again.
Denial
“No.. can’t be our stuff!”
Security prog
Something strange here!!!
Warning Will Robinson, warning.
You pay a fortune for a security prog and it gives you a warning? Esp. after an UPDATE!
Wonder if Any of them had an intermediary system Watching what was going back and forth OR one that they could use as a safe system, RUn everything First on a safe system to see what the security prog says.
Is that remotely practical? I worked for a software company, and one customer had some antivirus-related trouble. Just a performance problem, not flagging, but we never managed to get the attention of the antivirus people. We found some forum posts by other affected people explaining that it was related to long-running network connections, and nobody could get the company to fix it. So we all just changed our software to close and reopen connections periodically.
That was 15 years ago, and maybe things have improved. But I doubt it. If they were easy to deal with, people probably wouldn’t be so quick to add exceptions. The companies know that people are just buying their software because it’s been mandated.
Re:
“Is that remotely practical?”
Maybe not to talk to all of them, but it should be possible to contact a major vendor and confirm with them if there’s a false positive.
“The companies know that people are just buying their software because it’s been mandated.”
This is indeed a part of the problem, I think. Most people on the ops side of things hate AV software because of false positives and the performance overheads. Many people would also hate the 3CX stuff they have to run. Combine the two, a lot of people would rather be adding exceptions to what was working yesterday than actually deal with either of them, and that’s a problem. So much enterprise equipment is because a sales deal was made not because they’re the best tools for the people who have to run them.
“Instead, the 3CX rep said there were too many AV providers to go out there and call them all”
Indeed, but the fact that so many were flagging something should have been a clue that it wasn’t the virus checking that was the problem.
Although, I doubt the rep here understood what was going on. Some people were doubtless working desperately to work out the problem, but if it’s anything like some companies I’ve worked for the people you actually speak to on first line support were probably being told to make something up, if they were made aware of any issue at all. The described conversations sounds like what a non-tech savvy first line support person would say to get someone off the line rather than an official company statement.
“You really would think that after SolarWinds first and Kaseya second, tech companies would know better than to ignore this sort of thing and actually talk to the security firms that are flagging their products.”
Again, they might have been behind the scenes. But, there’s too many examples of companies that strip away real knowledge and stumble when something completely unexpected happens like this. Likely, there was some poor overworked individual who’s been fighting for a while for better security who was refused the resources while everything was “OK”
I have doubts. Your theory is plausible, but it’s also possible that this was an area outside of anyone’s responsibility. Programmers check in their code, and then it’s the responsibility of the build system administrators. A build came out, so their work is done, right? Then it goes to the testers, and antivirus programs are probably not part of their defined test environment. The bug reports come in to the support crew, who may not have the skills to reverse-engineer the problem (or the authority to install arbitrary antivirus software for testing); but they, like “everyone”, know that antivirus software can be fickle, so…
Customers who are big enough and loud enough will eventually be able to escalate the problem to someone who has the authority to make the necessary cross-department arrangements. Not every bug gets to this point. Mozilla’s open bug tracking system, for example, has stuff that’s been around for decades with people occasionally popping in to vote or comment, in unsuccessful attempts to get it fixed. The same happens in private corporate bug trackers.
Crappy companies are crappy
To add fuel to the funeral pyre for 3cx, their CEO is allegedly canceling contacts of people who complained too loudly or too forcefully. Also, and probably the best part, he claimed that this was due to an open source project’s file(s). Good for them that they immediately came back with the equivalent of “we don’t provide binaries so we’re not sure what hers on about.”