Researchers Who Built Similar System Explain Why Apple's CSAM Scanning System Is Dangerous

from the it's-not-good dept

Jonathan Mayer, a Princeton University professor and former chief technologist at the FTC, is one of the smartest people I know. Every time I’ve spoken with him I feel like I learn something. He’s now written a quite interesting article for the Washington Post noting how he, and a graduate researcher at Princeton, Anunay Kulshrestha, actually built a CSAM scanning system similar to the one that Apple recently announced, which has security experts up in arms over the risks inherent to the approach.

Mayer and Kulshretha note that while Apple is saying that people worried about their system are misunderstanding it, they are not. They know what they’re talking about — and they still say the system is dangerous.

We wrote the only peer-reviewed publication on how to build a system like Apple?s ? and we concluded the technology was dangerous. We?re not concerned because we misunderstand how Apple?s system works. The problem is, we understand exactly how it works.

Mayer and Kulshretha are certainly not making light of the challenges and problems associated with stopping CSAM. It’s why they started their project to see if there was an effective way to identify CSAM even in end-to-end encrypted systems. And they even built a system to do just that. And what they found is that the risks are simply too great.

We were so disturbed that we took a step we hadn?t seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides. We?d planned to discuss paths forward at an academic conference this month.

That dialogue never happened. The week before our presentation, Apple announced it would deploy its nearly identical system on iCloud Photos, which exists on more than 1.5 billion devices. Apple?s motivation, like ours, was to protect children. And its system was technically more efficient and capable than ours. But we were baffled to see that Apple had few answers for the hard questions we?d surfaced.

The potential dangers that Mayer and Kulshretha are exactly what many had warned about when Apple announced its plans:

After many false starts, we built a working prototype. But we encountered a glaring problem.

Our system could be easily repurposed for surveillance and censorship. The design wasn?t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser.

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That?s no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

That’s already pretty damning. But there’s some other even more damning information that has come out as well. As we noted in our earlier posts (and as Mayer and Kulshretha noted in their research), the risk of false positives is extremely high. And late last week, the hypothetical became a lot more real. Someone reverse engineered the NeuralHash algorithm that Apple is using and put it on Github.

It did not take long for people to point that, first of all, there are collisions with totally unrelated images:

Now, you might say that since that second image is not really an image, maybe it’s not that big a deal? Well, about that…

And, of course, one of the issues with any hash-based system is the idea that subtle changes to the image would output a totally different hash, thus making it easy for some to simply route around the scanning. Apple had suggested its system could defeat that, but…

This is part of the reason that we highlighted earlier that security researchers are so up in arms about this. Apple seemingly ignored so much of the research and conversations that were happening about these approaches, and just barged right in announcing that it had a solution without exploring the tradeoffs and difficulties associated with it — leaving that for security experts to point out afterwards.

Apple is trying to downplay these findings, saying that it expected the collisions at least, and that it’s system would also do a separate server side hashing comparison which would stop the false collisions. Though, as Bruce Schneier points out, if this was “expected,” then why wasn’t it discussed in the initial details that were released? Similarly, I have yet to see a response to the flip side issue of changing the images in a way that fool NeuralHash while still looking the same.

I know Apple keeps wanting to insist that it’s thought through all of this, but it doesn’t seem to have thought through any of how the security community would see all of this, and it’s after-the-fact scrambling is not exactly reassuring.

Filed Under: , , , , , ,
Companies: apple

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Researchers Who Built Similar System Explain Why Apple's CSAM Scanning System Is Dangerous”

Subscribe: RSS Leave a comment
This comment has been deemed insightful by the community.
Anonymous Coward says:

The big problem is that they have implemented a system, that has root access, and can exfiltrate anything they want from the phone. Scanning on the phone saves Apple CPU power, but is incidental to the fact that they have implemented a system that gives them full access and export for anything on the phone. That is not a backdoor, but rather full government access to phone contents, including keys as they are entered, as soon as a government can compel Apple to comply with their demands.

This comment has been deemed insightful by the community.
Anonymous Coward says:

Re: Re:

Quite the contrary. It is a backdoor, just one surrounded with neon lights, and illuminating arrows pointing right to it.

Full government access is one thing, but this goes much further than that. Apple will happily give access to anyone who’s willing to pay, so don’t be surprised it you’re called in front of your Employer’s HR because you have something on your iPhone they didn’t like.

Don’t count on "lawful-intercept" or wire-tapping laws to protect you either. If Apple hasn’t already, you can be sure they’ll updated iPhone terms and conditions, so they can claim that you have "voluntarily agreed" to this snooping.

Anonymous Coward says:

Apple says it won’t give in to GOVERNMENT demands.

Apple has already set up its own shell company. It plans to "sell" the data to the shell company, which will sell the data to various governments around the world.

Thus keeping "the literal letter of the promise, but actually making a lot of money".

The internal plans specify child porn as "the ice breaker" to get people used to being spied on, then they will start on emails, documents, passwords, browsing history one by one. A slow push back of privacy in the hopes the majority of customers won’t notice.

BG (profile) says:

Re: Re:

… and if official/government access was all we had to worry about that would be tolerable … possibly.

That’s not all we have to worry about though. A system like this is 100% going to be targeted by adversarial intelligence services, criminal, hackers, etc. One way or another people that have no legitimate or court authorised right to the system will make use of it. Whether it’s old fashioned bribery/blackmail, hacking, exploiting software flaws, etc. they will get access to it.

ECA (profile) says:

old idea

There was an interesting thought long ago, and a debate that didnt last to long.
If we all could read each others minds, wouldnt it be a great thing?
We could understand each other fully and with meaning.
And know every crook and thief around us.
Learning to control ourselves and our minds would be the first challenge.
We would know WHO needs help.
We could understand and KNow how others feel about Special circumstances, maybe find out how to Adjust them to be abit different. But we would know what created things in their minds.

But those most scared are the ones hiding truth.

This comment has been deemed insightful by the community.
Anonymous Coward says:


I think the folks at Apple may be enjoying a different kind of Hash if they can claim that they actually knew the problems pointed out and still went forward with it.

It’s a lot like replacing a bridge with nothing and pointing out the the cars were able to drive off a cliff without any problems and it was fully expected.

They should share that Hash.

That One Guy (profile) says:

'You had One Job'

A system for catching certain content that’s trivial to trick/bypass, will result in a significant number of false positives and can be easily repurposed to go after any number of other types of content without the user’s knowledge with the only thing stopping that being Apple pinky-promising not to do that and/or other groups(government or not) doing so themselves.

Oh yeah, they really thought this one thought before rolling it out.

Ninja (profile) says:

Considering it can be weaponized in many ways other than governments using it for censorship and the "for the children" mantra is often used by people that couldn’t care less about the children or civil rights (ie: the Trump horde, dictators around the world etc) this will be very ugly to watch. Thanks Apple. And if I didn’t already have enough reasons not to use Apple products now I have one that’s probably worth all possible reasons multiplied by a few orders of magnitude.

Darkness Of Course (profile) says:

Data center power usage model

Google showed up at conference and presented their whizzy solution. Reduce the transformation of DC to AC to DC, and some other bits.

Some got up in their face because there was an industry wide panel working on just that and they appeared to further down the road than Google.

If you are absolutely convinced you are the best, then checking your results against the work of others will never occur to you.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...