California AI Bill Tells GenAI Startups To Nerd Harder

Policy

from the this-is-something,-we-must-do-this dept

There’s a stunning degree of fear mongering and lack of humility about what California AI bill SB 942 can or can’t do. Honest conversation about this bill’s limitations are essential to ensuring we don’t pass this ineffective law. But its proponents have obstructed reasoned policy development by injecting panic into that conversation and pretending it will solve various multifaceted GenAI abuses that it simply cannot.

This article is a follow up to my first one, where I explain why SB 942’s forced disclosures and inclusion of AI-generated text were unworkable. Despite recent amendments that fix those two issues, the remaining requirement that nascent AI companies create “AI detection tools” make it still a fundamentally flawed bill.

SB 942 vaguely aims to “tackle the issue of GenAI-produced content” by requiring three things:

1. AI providers must offer a free AI detection tool that identifies content generated using their service. 
2. AI providers must offer users the option to include a conspicuous disclosure on content generated with their service. 
3. AI providers must embed specific metadata into files and the generated content itself, including company name, version number, timestamp, and a unique identifier.

A theme that runs through nearly every aspect of this bill’s journey through the legislature is a mismatch between any given GenAI abuse and how this bill would address it. For example, a June 28 committee analysis clumsily tries to explain the danger of open-source AI and incorrectly implies that SB 942 will help counter it: 

“ChatGPT is an example of an open-sourced tool, meaning it is accessible to the public. Researchers and developers can also access its code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes.”

While OpenAI has released some model weights, ChatGPT is famously not open-source. Here, open-source has been conflated with “widely available.” Ironically, this bill only applies to AI businesses, and not situations where actual open-source AI software may be abused. 

The analysis continues:

“The need for this bill is further highlighted by various instances and research underscoring the threats posed by unregulated GenAI.”

It goes on to describe three examples of GenAI abuse and again falsely implies this bill is the fix. Let’s talk about why that’s not true. 

Example 1:

In January, voters in New Hampshire received phone calls from an AI-generated voice clone of Joe Biden telling them not to vote in the primary election in order to save their vote for the upcoming general election. They caught the man responsible and he’s now facing a 6 million dollar fine and 13 felony charges. The FCC has also now made AI robocalls illegal in response.  

The bill’s three main provisions would not prevent this from happening again: 1) It would be impractical for consumers to record robocalls and then upload them to a detection tool. 2) Nefarious actors would opt out of the optional disclosures and 3) metadata for the fraudulent audiofile would be irrelevant in the context of an ephemeral phone call. 

Example 2:

A finance worker at a Hong Kong firm was persuaded to transfer $25 million to thieves when they were invited to participate in a video call with several other AI-generated colleagues.

This case shares the same pitfalls as the previous example: 1) An employee who thinks they may be in the middle of a fraudulent video call won’t have any way to upload that information to a detection tool during that session. 2) Bad actors will opt out of disclosures and 3) metadata will again be irrelevant.  

Example 3:  

California high school students have been caught generating nonconsensual nude images of their classmates.

1) Determining the authenticity of this content is useless and doesn’t address the harm it creates. 2) Even if a user opts for a disclosure, the content is still extremely harmful. 3) A hidden disclosure embedded into content might be helpful only in cases where no other evidence or context leads to the perpetrator. But this assumes that all companies can implement this still-developing idea across audio, video, and images. Requiring by law that this be implemented by startups is heavy handed and unrealistic.      

The analysis ends with a hypothetical that highlights yet another fundamental flaw:

“In theory, a person who views a video circulating on social media conveying President Joe Biden telling voters not to vote in the primary election could use a provider’s AI detection tool to upload and analyze the video. By examining the embedded machine-readable disclosures, the user could identify the provider’s name, the GenAI system used, and the creation date, concluding that the video was produced by AI. This process would reveal that the video is not genuine, thus, in theory, helping to prevent the spread of misinformation.”

That sounds nice, in theory. But in reality, it’s more complicated than that.

A social media user that uploads to an AI detection tool may not be returned any useful indication about a piece of content’s authenticity because the law only requires AI providers to determine if their service was used. The user would then need to make uploads to every AI provider’s tool until they get a hit, or give up before they do. The user will also need to understand that each tool has a different level of accuracy across video, image, and audio, and will need to account for the fact that there may be false positive and negative results. 

One alternative to this rigmarole is for users to do what they’ve always done when inquiring about suspicious online content: google it. 

But seriously, how many different AI detection websites will one have to potentially go to? Is it 5? 15? Maybe 50? I doubt anyone has contemplated this number. AI providers with over 1M monthly users will have to comply with this law, making the user threshold arbitrarily over inclusive given that there are at least 1,500 GenAI startups poised for growth. And the unlucky startups that already have 1M users will have only four months to develop this unproven technology before the law takes effect in January of next year.

By passing this law, California lawmakers would be telling an untold number of GenAI startups to nerd harder. 

And now we get to the weird part. 

In the July 2 hearing [2:55hr mark] for which our much-discussed committee analysis was prepared, co-drafter of the bill Tom Kemp again testified in support. 

One statement stood out in particular:

“Recent changes to the bill have addressed many opposition concerns coming out of the privacy committee. For example, a critic just recently wrote that the recent changes have quote, ‘fixed the biggest issues I’ve had with the bill.’ ” 

Who is this unnamed critic? Well, Kemp appears to quote a comment that I made while summarizing several amendments in a Linkedin post. Read next to my own commentary, the two phrases are almost identical:

“These changes fix the biggest issues I had with the bill, link in comments.”

These two phrases don’t appear anywhere else on the internet, suggesting Kemp did in fact quote me. That’s a shame because while amendments did fix two of three issues I originally pointed out, this short phrasing doesn’t reflect my full thoughts on the bill. SB 942, in my opinion, is still a hot mess. 

The amendments didn’t address the issue of compelled AI detection tools. As we’ve already discussed, requiring AI providers to create detection tools does not guarantee them to actually work well. The fact that we can’t rely on them 100% of the time means there will be false positives and negatives, which has already proved to be highly damaging. Mandating these tools by law, with no consideration for how inaccurate they may be, is tech solutionism and will only confuse people more about the authenticity of content they encounter. 

I regret that my choice of a few short words was used to promote this piece of legislation. To avoid any doubt, I’ve edited my Linkedin post to read:

“These changes fix [some of] the biggest issues I had with the bill, [but it still has many, many issues.]“

Alan Kyle is a tech policy professional available for hire in AI Governance, Trust & Safety, and Privacy.

Filed Under: california, generative ai, josh becker, nerd harder, sb 942, tom kemp

UK Government ‘Concession’ On Breaking End-to-End Encryption In The Online Safety Act (Just Passed) Turns Out Not To Be One

Privacy

from the Schrödinger's-encryption-backdoor dept

Last week Techdirt wrote about an important development in the long-running saga of the UK’s Online Safety Act, which has just become law. The UK government said at that time it would not use controversial powers in the new law to break end-to-end encryption until it was “technically feasible” to do so while preserving users’ privacy. That seemed to be a recognition that it was impossible to carry out scanning that safeguarded privacy with any existing technology, despite previous claims to the contrary. Since it is extremely unlikely such technology will ever exist, the hope was that the UK government was effectively dropping the idea with this concession. But in the days that followed, this optimistic interpretation has seemed less certain. When the “technically feasible” caveat was first mentioned, the Guardian pointed out:

the government has not changed the wording of the bill, which still gives [the UK regulatory body] Ofcom the power to issue an accredited technology notice. A government spokesperson said: “Our position on this matter has not changed”.

Further evidence that the underlying intent hasn’t changed is found in an article in the Independent:

[UK] Technology Secretary Michelle Donelan insisted that nothing had changed in the long-awaited legislation, after privacy campaigners earlier this month claimed a victory following widespread reports of a shift in the Government stance on encryption.

Donelan gave more details of how the new Online Safety Act would work in practice:

In terms of end-to-end encryption, when a platform about to encrypt or already has encrypted – if there were concerns then raised with the regulator that there was paedophilia or child abuse on there, then the regulator would have a conversation with that platform, see what mitigations they could put in place to adhere to the legislation.

If none of that worked, we need a safety net built into this piece of legislation – and the safety net works by the regulator saying you now need to invest in technology that will allow you to maintain the privacy element of encryption, protect encryption, but also enable us to have access and find these criminals, these heinous individuals, these paedophiles, these stains on society.

It may never have to be used. But we think it is important that we put that safety net in legislation.

So it seems the UK government’s idea is that Internet companies will be ordered to come up with ways to break end-to-end encryption while maintaining privacy. But don’t worry, because that magic encryption backdoor will only be there as a “safety net”, not as something that will ever be used routinely. Of course.

Once again, the UK government is attempting an impossible balancing act. On the one hand, it needs to keep the extreme wing of its party happy by bringing in surveillance of encrypted communications. On the other, it doesn’t want the UK to lose key messaging services like Signal, WhatsApp and iMessage, which have all said they won’t implement back doors. Its solution seems to be the usual demand that tech companies “nerd harder”, plus a promise that the new surveillance powers would only be used if the “mitigations” don’t work.

The hardliners who don’t understand the technology might be happy with that approach, but the tech companies won’t be. As soon as the latter are ordered to begin that harder nerding, they will probably pull out of the UK. In other words, despite the “technically feasible” fig leaf, nothing has changed. The UK government’s desperate attempt to come up with Schrödinger’s encryption backdoor – there for the police, but not there for the tech companies – has failed. It had to choose between mass surveillance and messaging services; by passing the Online Safety Act with the text unchanged, it seems to have chosen surveillance.

Follow me @glynmoody on Mastodon.

Filed Under: backdoors, concession, end-to-end encryption, imessage, nerd harder, ofcom, online safety bill, Schrödinger, signal, uk, whatsapp

Congress’ Kids Online Safety Act Puts Kids At Risk Through Fuzzy Language

Policy

from the will-this-protect-or-harm-children? dept

The Kids Online Safety Act (KOSA) was voted out of committee with a long list of amendments. Advocates had been warning about some severe unintended consequences that could arise out of this bill, the most concerning of which was forcing tech companies to out LGBTQ+ minors to their parents — potentially against their wishes. The amendments were supposed to fix these issues and more. But did they?

The short answer is there was an honest attempt but I believe it falls short, and I think it falls short for a specific reason.

The background of the bill

In order to understand why this bill has significant problems, we first have to cover some basics and separate the intended and unintended harms from the bill. 

Let’s start with what the bill wants to do, which is set a floor of protections for minors. It does that by creating a duty of care to act in the best interest of the minor. The bill then goes on to loosely define what that means and what category of harms online platforms need to be shielding minors against, requiring the creation of certain tools parents can use to monitor their kids, etc.. It also gives platforms plenty of homework, like creating an annual report identifying the risks they think minors will encounter on their platform and what they are doing to mitigate those harms.

So why did I say this bill has intended harms? Well drafting a bill is hard, you have to describe what you mean when you say a company “shall act in the best interests of a minor” to “take reasonable measures” to “prevent and mitigate mental health disorders” or “addiction”. The more granular you get the more confusing it gets and the more broad it’s stated the harder it is to apply to specific facts. 

Let’s say I’m playing a game with VOIP and someone calls me a slur. Was that because the game company failed to take reasonable measures? If I want to play a game during all my free time is it because the game is really good or because it was intentionally made to provoke “compulsive usage”? What even are “reasonable measures”? Especially when many of the things the bill describes impacts people differently.

KOSA’s intentional fuzzy language

KOSA’s authors are basically outsourcing to courts how to apply the bill’s fuzzy language to actual facts. Practically, this means that if the bill is passed all platforms will attempt to comply with what they think the text means. Then at least one of the platforms will almost certainly get sued for falling short. Those companies will then have to go through a lot of discovery and judges will just muddle through it. 

This will be a lengthy, painful, expensive, and time consuming process. But I think it’s intentional. Many in Congress think that platforms are not doing enough to protect kids, even though they should have the resources to do so. They either don’t see, or don’t care about, the large amount of resources already going into trust and safety divisions to protect all users, including minors. They see a problem that needs to be immediately solved, and believe a strong regulatory response will give platforms enough of a kick in the pants to figure it out. This is the famous “nerd harder” complaint that often gets leveled at Silicon Valley.

If you look at KOSA through this lens, everything kind of makes sense. It doesn’t matter that it sets up a bunch of expensive new compliance efforts that may or may not be productive. It doesn’t matter that it may kill off some companies or force consolidation. It doesn’t even matter that some platforms will try to bar minors from their platform completely (of course we all know that kids will figure out how to get on the platforms anyways). It’s a big extrinsic shock that they hope will shake things up enough so that platforms will finally nerd hard enough.

After all, the enforcement of KOSA is limited to the FTC and state AGs. We can trust them to only bring cases that will advance the welfare of children right?

KOSA’s extremely bad unintended harms

In Normal Times™, this is how the debate on whether to pass KOSA would go: this bill is a mess and will be too painful (and expensive) to sort out — vs. — we really don’t care, the platforms can afford it, and we think it will do something to at least make the world slightly better.

But these aren’t normal times, and advocates have been warning that not only will this bill be painful to sort out, it provides an avenue of attack from ideologues using the legal system to go after marginalized communities. This is a real threat that no lawmaker (especially Democrats) should be complicit in, especially considering that the overturning of Roe has become a starter pistol for using the legal system towards culture wars and extreme ideological ends.

The main avenue of attack built into the original KOSA was towards the LGBTQ community, and the feedback given at the time was that it will out kids to parents that might not be tolerant and could result in things like minors being thrown out of homes or sent to conversion therapy. This is what advocates warned the drafters of, and what new language sought to fix.

So was this fixed? Sort of. They added a provision saying that the bill shouldn’t be interpreted to require the disclosure to parents of things like browsing behavior, search history, messages, content of communications. The tools that platforms are required to provide to parents now seems solely directed at high level things like time used, purchases, etc.. But, there is a sort of dangling requirement that there are “control options that allow parents to address the harms described in” the big section describing the harms they want to stop. What option stops bullying? I’d like to know (maybe it will allow me to stop being T-bagged in multiplayer games).

Sorting that out may or may not sweep some sensitive data back in and expose kids. Sometimes kids keep secrets to protect themselves from their parents. This makes sense to me, I had a friend growing up sent to one of the reform schools Paris Hilton warned us about. However, I’m overall less concerned about forced outing than I was before the amendments.

I’m now more concerned this bill invites a broad attack against platforms allowing a kid to see any pro-LGBTQ content. The culture wars’ Eye of Sauron has turned to harassment and vile behavior towards this community, especially trans persons, and they are doing so under the banner of protecting children.

Unfortunately, the language these people are using to vilify the LGBTQ community is everywhere in the bill. Being trans has been called a mental health disorder, and this bill says platforms are required to protect minors from that. Seeing a drag queen, period, has also been described as sexual exploitation, grooming, and sexual abuse. Again, barred in KOSA. Gender affirming care has been referred to as self-harm, which again platforms are required to protect against under KOSA.

The bill’s fuzzy language, which may have been seen by the drafters as an asset, is now a huge liability. And it’s not just limited to anti-LGBTQ content. For example, a minor seeking information about how to receive a safe abortion could also be described as self-harm.

The bill’s authors might think that they are safe from their bill being used in these culture wars because enforcement is limited to the FTC and State Attorneys General. While I worry less about the FTC (now) it’s easy to imagine certain state AGs getting before the right judge and successfully barring minors from access to basic information they need to understand what they are going through and how to receive help, if they need it. Just look to Florida, where governor Desantis has filed a complaint against a restaurant and bar that allowed kids at a drag brunch and said that parents that allow their children to see a drag performance could be targeted by child protective services. 

This bill is throwing a hand grenade into the middle of a particularly dark moment of our legal system. I don’t think that’s wise, or very smart politically when the odds are actually quite high someone decides to take this bill up on its offer. 

Matthew Lane is a Senior Director at InSight Public Affairs.

Filed Under: congress, fuzzy language, kosa, lgbtq, nerd harder, parents, protect the children, secrecy, trust and safety

This Week Only: Free Shipping On Techdirt Gear From Threadless

Deals

from the get-it-quick dept

Get free shipping on Techdirt Gear orders over $45 with the coupon code FREESHIP92031e946 »

Have you had your eye on some gear from the Techdirt store on Threadless? Then this is the week to pick it up! From now until Friday at 3pm PDT, you can get free shipping on orders over $45 in the US and $80 international with the coupon code FREESHIP92031e946. The offer covers all our designs, including the new Otherwise Objectionable gear celebrating two of the most important words in Section 230, and our wide variety of face masks.

There’s also our complete line of Techdirt logo gear and, as usual, a wide variety of products available in every design: t-shirts, hoodies, sweaters and other apparel — plus a variety of cool accessories and home items including buttons, phone cases (for many iPhone and Galaxy models), mugs, tote bags, and stylish notebooks and journals.

This week only! Get free shipping with the coupon code FREESHIP92031e946 »

Filed Under: emojiment, gear, nerd harder, sale, techdirt gear

Masks & More: Techdirt Logo Gear Is Now On Threadless

Techdirt

from the wear-your-techdirt dept

Get your Techdirt Logo Gear in our store on Threadless »

Lots of people have been buying our newly-launched face masks on Threadless, so we figured it was time to make the classic Techdirt logo gear available on the platform. You can now get face masks in both our logo gear styles — the standard logo and the big logo — with two different kinds of masks available, as well as youth sizes.

Plus, as with all our designs, there’s a wide variety of gear available: t-shirts, hoodies, sweaters and other apparel — plus cool accessories and home items including buttons, phone cases (for many iPhone and Galaxy models), mugs, tote bags, and stylish notebooks and journals. And you can still get all our other popular designs:

As always, all the profits from gear sales help us keep Techdirt going and continue our reporting through this challenging pandemic situation and beyond. You can also check out our list of all the different ways to support Techdirt with a bunch of options for readers to help us out and get something cool or useful in return!

Filed Under: gear, merchandise, nerd harder, ok landlord, techdirt, techdirt gear

In Case You Missed It: The Return Of Nerd Harder Gear, Plus New Face Masks!

Techdirt

from the mask-harder dept

Nerd Harder gear is back, and face masks are available
in the Techdirt Gear store on Threadless »

At the end of last month, we fulfilled two of the most popular requests (one long-standing, and one brand new) for Techdirt gear: we brought back the Nerd Harder line of gear, and introduced a series of face masks featuring some of our most popular designs!

You can find all these offerings in our artist store on Threadless, with multiple products available. Face masks come in two styles (standard and premium) as well as youth sizes, and there are t-shirts, hoodies, sweaters and other apparel — plus a variety of cool accessories and home items including buttons, phone cases (for many iPhone and Galaxy models), mugs, tote bags, and stylish notebooks and journals.

All the profits from gear sales help us keep Techdirt going and continue our reporting through this challenging pandemic situation and beyond, and we’re hugely appreciative of all the support. You can also check out our list of all the different ways to support Techdirt with a wide variety of options for readers to help us out and get something cool or useful in return!

Filed Under: gear, merchandise, nerd harder, ok landlord, techdirt, techdirt gear

Techdirt Gear: New Masks, Old Favorites

Techdirt

from the face-it dept

Face masks are now available in the Techdirt store on Threadless »

We’ve had some requests for this, and now it begins: we’re adding face masks as an option for all our gear in the Techdirt store on Threadless. Not just that, we’ve expanded the Threadless line with our ever-popular Takedown gear and the much-anticipated return of our most successful design ever: Nerd Harder.

All the face masks are available in two versions (premium and standard) as well as youth sizes. And of course, the designs are also available on a wide variety of other products including t-shirts, hoodies, mugs, buttons, and more! Check out the Techdirt store on Threadless and order yours today.

Filed Under: face masks, nerd harder, ok landlord, takedown, techdirt, techdirt gear

Encryption Working Group Releases Paper To 'Move The Conversation Forward'

(Mis)Uses of Technology

from the what-conversation? dept

One of the frustrating aspects of the “debate” (if you can call it that) over encryption and whether or not law enforcement should be able to have any kind of “access” is that it’s been no debate at all. You have people who understand encryption who keep pointing out that what is being asked of them is impossible to do without jeopardizing some fairly fundamental security principles, and then a bunch of folks who respond with “well, just nerd harder.” There have been a few people who have suggested, at the very least, that “a conversation” was necessary between the different viewpoints, but mostly when that’s brought up it has meant non-technical law enforcement folks lecturing tech folks on why “lawful access” to encryption is necessary.

However, it appears that the folks at the Carnegie Endowment put together an actual working group of experts with very varying viewpoints to see if there was any sort of consensus or any way to move an actual conversation forward. I know or have met nearly everyone on the working group, and it’s an impressive group of very smart, and thoughtful people — even those I frequently disagree with. It’s a really good group and the paper they’ve now come out with is well worth reading. I don’t know that it actually moves the conversation “forward” because, again, I’m not sure there is any conversation to move forward. But I do appreciate that it got past the usual talking points. The paper kicks off by saying that it’s going to “reject two straw men,” which are basically the two positions frequently stated regarding law enforcement access to encrypted communication:

First of all, we reject two straw men?absolutist positions not actually held by serious participants, but sometimes used as caricatures of opponents?(1) that we should stop seeking approaches to enable access to encrypted information; or (2) that law enforcement will be unable to protect the public unless it can obtain access to all encrypted data through lawful process. We believe it is time to abandon these and other such straw men.

And… that’s fine, in that the first of those statements is not actually the position those who support strong encryption actually hold. I mean, there have been multiple reports detailing how we’re actually in the “golden age of surveillance”, and that law enforcement has so much greater access to basically every bit of communications possible, and that there are plenty of tools and ways to get information that is otherwise encrypted. Yes, it’s true that some information might remain encrypted, but no one has said that law enforcement shouldn’t do their basic detective work in trying to access information. The argument is just that they shouldn’t undermine the basic encryption that protects us all to do so.

Where the paper gets perhaps more interesting is that it suggests that any debate about access to encrypted data should focus on “data at rest” (i.e., data that is encrypted on a device) rather than “data in motion” which is the data that is being transferred across a network or between devices in some form. The paper does not say that we should poke holes in encryption that protects data at rest, and says, explicitly:

We have not concluded that any existing proposal in this area is viable, that any future such proposals will ultimately prove viable, or that policy changes are advisable at this time

However, it does note that if there is a fruitful conversation on this topic, it’s likely to be around data at rest, rather than elsewhere. And, from there it notes that any discussion of proposals for accessing such data at rest must take into account both the costs and the benefits of such access to determine if it is viable. While some of us strongly believe that there is unlikely to ever be a proposal where the costs don’t massively outweigh the benefits, this is the correct framework for analyzing theses things. And it should be noted that, too often, these debates involve one group only talking about the benefits and another only talking about the costs. Having a fruitful discussion requires being willing to measure both.

From there, the group sets up a framework for how to weigh those costs and benefits — including setting up a bunch of use cases against which any proposal should be tested. Again, this seems like the right approach to systematically exploring and stress testing any idea brought forth that claims it will “solve” the “problem” that some in law enforcement insist encryption has created for them. I am extremely skeptical that any such proposal can pass such a stress test in a manner that suggests that the benefits outweigh the costs — but if those pushing to undermine encryption require a “conversation” and want people to explore the few proposals that have been brought up, this is the proper, and rigorous, way to do so.

The question, though, remains as to whether or not this will actually “move the conversation forward.” I have my doubts on that, in part because those who keep pressing for undermining encryption have never appeared to have much interest in actually having this type of conversation. They have mostly only seemed interested in the “nerd harder, nerds” approach to this, that assumes smart techies will give them their magic key without undermining everything else that keeps us secure. I fully expect that it won’t be long before a Willam Barr or Chris Wray or a Richard Burr or a Cy Vance starts talking nonsense again about “going dark” or “responsible encryption” and ignores the framework set out by this working group.

That’s not so say this wasn’t a useful exercise. It likely was, if only to be able to point to it the next time one of the folks listed above spout off again as if there are no tradeoffs and as if it’s somehow easy to solve the “encryption problem” as they see it.

Filed Under: data at rest, encryption, going dark, law enforcement, nerd harder, responsible encryption

Software Legend Ray Ozzie Thinks He Can Safely Backdoor Encryption; He's Very Wrong

(Mis)Uses of Technology

from the and-dangerous dept

There have been ongoing debates for a while now about the stupidity of backdooring encryption, with plenty of experts explaining why there’s no feasible way to do it without causing all sorts of serious consequences (some more unintended than others). Without getting too deep into the weeds, the basic issue is that cryptography is freaking difficult and if something goes wrong, you’re in a lot of trouble very fast. And it’s very, very easy for something to go wrong. Adding in a backdoor to encryption is, effectively, making something go wrong… on purpose. In doing so, however, you’re introducing a whole host of other opportunities for many, many things to go wrong, blowing up the whole scheme and putting everyone’s information at risk. So, if you’re going to show up with a “plan” to backdoor encryption, you better have a pretty convincing argument for how you avoid that issue (because the reality is you can’t).

For at least a year (probably more) the one name that has kept coming up over and over as one of the few techies who insists that the common wisdom on backdooring encryption is wrong… is Ray Ozzie. Everyone notes that he’s Microsoft’s former Chief Software Architect and CTO, but some of us remember him from way before that when he created Lotus Notes and Groove Networks (which was supposed to be the nirvana of collaboration software). In recent months his name has popped up here and there, often by FBI/DOJ folks seeking to backdoor encryption, as having some possible ways forward.

And, recently, Wired did a big story on his backdoor idea, where he plays right into the FBI’s “nerd harder” trope, by saying exactly what the FBI wants to hear, and which nearly every actual security expert says is wrong:

Ozzie, trim and vigorous at 62, acknowledged off the bat that he was dealing with a polarizing issue. The cryptographic and civil liberties community argued that solving the problem was virtually impossible, which ?kind of bothers me,? he said. ?In engineering if you think hard enough, you can come up with a solution.? He believed he had one.

This, of course, is the same sort of thing that James Comey, Christopher Wray and Rod Rosenstein have all suggested in the past few years: “you techies are smart, if you just nerd harder, you’ll solve the problem.” Ozzie, tragically, is giving them ammo. But he’s not delivering the actual goods.

The Wired story details his plan which is not particularly unique. It takes concepts that others have proposed (and which have been shown to not be particularly secure) and puts a fresh coat of paint on them. Basically, the vendor of a device has a private key that it needs to keep secret, and under some “very special circumstances” it can send an employee into the dark chamber to do the requisite dance, retrieve the code, and give it to law enforcement. That’s been suggested many times, and it’s been explained many times why that opens up all sorts of dangerous scenarios that could put everyone at risk. The one piece that does seem different is that Ozzie wants a sort of limitation on the possible damage his system does if it goes wrong (in one particular way), which is that under his system if the backdoor is used, it can only be used on one phone and then it disables that phone forever:

Ozzie designed other features meant to ?reassure skeptics. Clear works on only one device at a time: Obtaining one phone?s PIN would not give the authorities the means to crack anyone else?s phone. Also, when a phone is unlocked with Clear, a special chip inside the phone blows itself up, freezing the contents of the phone thereafter. This prevents any tampering with the contents of the phone. Clear can?t be used for ongoing surveillance, Ozzie told the Columbia group, because once it is employed, the phone would no longer be able to be used.

So, let’s be clear. That piece isn’t what’s useful in “reassuring skeptics.” That piece is the only thing that really appears to be that unique about Ozzie’s plan. And it hasn’t done much to reassure skeptics. As the report notes, when Ozzie laid this out at a special meeting of super smart folks in the field, it didn’t take long for one to spot a hole:

The most dramatic comment came from computer science professor and cryptographer Eran Tromer. With the flair of Hercule Poirot revealing the murderer, he announced that he?d discovered a weakness. He spun a wild scenario involving a stolen phone, a second hacked phone, and a bank robbery. Ozzie conceded that Tromer found a flaw, but not one that couldn?t be fixed.

“Not one that couldn’t be fixed.” But it took this guy just hearing about the system to find the flaw. There are more flaws. And they’re going to be catastrophic. Because that’s how cryptogrpahy works. Columbia computer science professor and all around computer security genius Steve Bellovin (who was also at that meeting) highlights how Tromer’s flaw-spotting shows why Ozzie’s plan is a fantasy with dangerous consequences:

Ozzie presented his proposal at a meeting at Columbia?I was there?to a diverse group. Levy wrote that Ozzie felt that he had “taken another baby step in what is now a two-years-and-counting quest” and that “he’d started to change the debate about how best to balance privacy and law enforcement access”. I don’t agree. In fact, I think that one can draw the opposite conclusion.

At the meeting, Eran Tromer found a flaw in Ozzie’s scheme: under certain circumstances, an attacker can get an arbitrary phone unlocked. That in itself is interesting, but to me the important thing is that a flaw was found. Ozzie has been presenting his scheme for quite some time. I first heard it last May, at a meeting with several brand-name cryptographers in the audience. No one spotted the flaw. At the January meeting, though, Eran squinted at it and looked at it sideways?and in real-time he found a problem that everyone else had missed. Are there other problems lurking? I wouldn’t be even slightly surprised. As I keep saying, cryptographic protocols are hard.

Bellovin also points out — as others have before — that there’s a wider problem here: how other countries will use whatever stupid example the US sets for much more nefarious purposes:

If the United States adopts this scheme, other countries, including specifically Russia and China, are sure to follow. Would they consent to a scheme that relied on the cooperation of an American company, and with keys stored in the U.S.? Almost certainly not. Now: would the U.S. be content with phones unlockable only with the consent and cooperation of Russian or Chinese companies? I can’t see that, either. Maybe there’s a solution, maybe not?but the proposal is silent on the issue.

And we’re just getting started on how many experts are weighing in on just how wrong Ozzie is. Errata Security’s Rob Graham pulls no punches pointing out that:

He’s only solving the part we already know how to solve. He’s deliberately ignoring the stuff we don’t know how to solve. We know how to make backdoors, we just don’t know how to secure them.

Specifically, Ozzie’s plan relies on the idea that companies can keep their master private key safe. To support that this is possible, Ozzie (as the FBI has in the past) points to the fact that companies like Apple already keep their signing keys secret. And that’s true. But that assumes incorrectly that signing keys and decryption keys are the same thing and can be treated similarly. They’re not and they cannot be. The security protocols around signing keys are intense, but part of that intensity is built around the idea that you almost never have to use a signing key.

A decryption key is a different story altogether, especially with the FBI blathering on about thousands of phones it wants to dig its digital hands into. And, as Graham notes, you quickly run into a scaling issue, and with that scale, you ruin any chance of keeping that key secure.

Yes, Apple has a vault where they’ve successfully protected important keys. No, it doesn’t mean this vault scales. The more people and the more often you have to touch the vault, the less secure it becomes. We are talking thousands of requests per day from 100,000 different law enforcement agencies around the world. We are unlikely to protect this against incompetence and mistakes. We are definitely unable to secure this against deliberate attack.

And, even worse, when that happened, we wouldn’t even know.

If Ozzie’s master key were stolen, nothing would happen. Nobody would know, and evildoers would be able to freely decrypt phones. Ozzie claims his scheme can work because SSL works — but then his scheme includes none of the many protections necessary to make SSL work.

What I’m trying to show here is that in a lab, it all looks nice and pretty, but when attacked at scale, things break down — quickly. We have so much experience with failure at scale that we can judge Ozzie’s scheme as woefully incomplete. It’s not even up to the standard of SSL, and we have a long list of SSL problems.

And so Ozzie’s scheme relies on an impossibility. That you could protect a decryption key that has to be used frequently, the same way that a signing key is currently protected. And that doesn’t work. And when it fails, everyone is seriously fucked.

Graham’s article also notes that Ozzis is — in true nerd harder fashion — focusing on this as a technological problem, ignoring all the human reasons why such a system will fail and such a key won’t be protected.

It focuses on the mathematical model but ignores the human element. We already know how to solve the mathematical problem in a hundred different ways. The part we don’t know how to secure is the human element.

How do we know the law enforcement person is who they say they are? How do we know the “trusted Apple employee” can’t be bribed? How can the law enforcement agent communicate securely with the Apple employee?

You think these things are theoretical, but they aren’t.

Cryptography expert (and professor at Johns Hopkins), Matt Green did a fairly thorough tweetstorm debunking of Ozzie’s plan as well. He also points out, as Graham does, the disaster scenario of what happens when (not if) the key gets out. But, an even bigger point that Green makes is that Ozzie’s plan relies on a special chip in every device… and assumes that we’ll design that chip to work perfectly and never get broken. And that’s ridiculous:

Green and Graham also both point to the example of GrayKey, the recently reported on tool that law enforcement has been using to crack into all supposedly encrypted iPhones. Already, someone has hacked into the company behind GrayKey and leaked some of the code.

Put it all together and:

Suddenly the fawning over Ozzie’s plan doesn’t look so good any more, does it? And, again, these are the problems that everyone who has dug into why backdoors are a bad idea have pointed out before:

Green expanded some of his tweets into a blog post as well, which is also worth reading. In it, he also points out that even if we acknowledge the difference between signing keys and decryption keys, companies aren’t even that good at keeping signing keys safe (and those are almost certainly going to be more protected that decryption keys since they need to be access much less frequently):

Moreover, signing keys leak all the time. The phenomenon is so common that journalists have given it a name: it?s called ?Stuxnet-style code signing?. The name derives from the fact that the Stuxnet malware ? the nation-state malware used to sabotage Iran?s nuclear program ? was authenticated with valid code signing keys, many of which were (presumably) stolen from various software vendors. This practice hasn?t remained with nation states, unfortunately, and has now become common in retail malware.

And he also digs deeper into the point he made in his tweetstorm about how on the processor side, not even Apple has been able to keep its secure chip from being broken — yet Ozzie’s plan is based almost entirely on the idea that such an unbreakable chip would be available:

The richest and most sophisticated phone manufacturer in the entire world tried to build a processor that achieved goals similar to those Ozzie requires. And as of April 2018, after five years of trying, they have been unable to achieve this goal ? a goal that is critical to the security of the Ozzie proposal as I understand it.

Now obviously the lack of a secure processor today doesn?t mean such a processor will never exist. However, let me propose a general rule: if your proposal fundamentally relies on a secure lock that nobody can ever break, then it?s on you to show me how to build that lock.

Update: We should add that the criticisms raised here are not new either. Back in February we wrote about a whitepaper by Riana Pfefferkorn making basically all of these same points that the folks quoted above are making. In other words, it’s a bit bizarre that Wired wrote this article as if Ozzie is doing something new and noteworthy.

So that’s a bunch of experts highlighting why Ozzie’s plan is silly. But, from the policy side it’s awful too. Because having Ozzie going around and spouting this debunked nonsense, but with his pedigree, simply gives the “going dark” and “responsible encryption” pundits something to grasp onto to claim they were right all along, even though they weren’t. They’ve said for years that the techies just need to nerd harder, and they will canonize Ray Ozzie as the proof that they were right… even though they’re not and his plan doesn’t solve any of the really hard problems.

And, as we noted much earlier in this post, cryptography is one of those areas where the hard problems really fucking matter. And if Ozzie’s plan doesn’t even touch on most of the big ones, it’s no plan at all. It’s a Potemkin Village that law enforcement types will parade around for the next couple of years insisting that backdoors can be made safely, even though Ozzie’s plan is not safe at all. I am sure that Ray Ozzie means well — and I’ve got tremendous respect for him and have for years. But what he’s doing here is actively harmful — even if his plan is never implemented. Giving the James Comeys and Chris Wrays of the worlds some facade they can cling to to say that this can be done is only going to create many more problems.

Filed Under: encryption, encryption is hard, going dark, key escrow, matthew green, nerd harder, ray ozzie, responsible encryption, rob graham, security, steven bellovin

DOJ Continues Its Push For Encryption Backdoors With Even Worse Arguments

Say That Again

from the let-us-save-you-from-your-security dept

Early last week, the Deputy Attorney General (Rod Rosenstein) picked up the recently-departed James Comey’s Torch of Encroaching Darkness +1 and delivered one of the worst speeches against encryption ever delivered outside of the UK.

Rosenstein apparently has decided UK government officials shouldn’t have a monopoly on horrendous anti-encryption arguments. Saddling up his one-trick pony, the DAG dumped out a whole lot of nonsensical words in front of a slightly more receptive audience. Speaking at the Global Cyber Security Summit in London, Rosenstein continued his crusade against encryption using counterintuitive arguments.

After name-dropping his newly-minted term — responsible encryption™ — Rosenstein stepped back to assess the overall cybersecurity situation. In short, it is awful. Worse, perhaps, than Rosenstein’s own arguments. Between the inadvertently NSA-backed WannaCry ransomware, the Kehlios botnet, dozens of ill-mannered state actors, and everything else happening seemingly all at once, the world’s computer users could obviously use all the security they can get.

Encryption is key to security. Rosenstein agrees… up to a point. He wants better security for everyone, unless those everyones are targeted by search warrants. Then they have too much encryption.

Encryption is essential. It is a foundational element of data security and authentication. It is central to the growth and flourishing of the digital economy. We in law enforcement have no desire to undermine encryption.

But “warrant-proof” encryption poses a serious problem.

Well, you can’t really have both secure encryption and law enforcement-friendly encryption. Rosenstein knows this just as surely as Comey knew it. That didn’t stop Comey from pretending it was all about tech company recalcitrance. The same goes for Rosenstein who, early on in his speech, plays a shitty version of Sympathy for the Tech Devil by using the phrase “competitive forces” as a stand-in for “profit seeking” when speaking about the uptick in default encryption.

The underlying message of his last speech was that American tech companies should spurn profits for helping out the government by unwrapping one end of end-to-end encryption. The same pitch is made here, softened slightly in the lede thanks to the presence of UK tech companies in the audience. The language may be less divisive, but the arguments are no less stupid this time around.

In the United States, when crime is afoot, impartial judges are responsible for balancing a citizen’s reasonable expectation of privacy against the interests of law enforcement. The law recognizes that legitimate law enforcement needs can outweigh personal privacy concerns. That is how we obtain search warrants for homes and court orders to require witnesses to testify.

Warrant-proof encryption overrides our ability to balance privacy and security. Our society has never had a system where evidence of criminal wrongdoing was impervious to detection by officers acting with a court-authorized warrant. But that is the world that technology companies are creating.

I’m not sure what this “system” is Rosenstein speaks about, but there has always been evidence that’s eluded the grasp of law enforcement. Prior to common telephone use, people still communicated criminal plans but no one insisted citizens hold every conversation within earshot of law enforcement. Even in a digital world, evidence production isn’t guaranteed, even when encryption isn’t a factor.

Going on from there, the rest of speech is pretty much identical to his earlier one. In other words: really, really bad and really, really wrong.

Rosenstein believes the government should be able to place its finger on the privacy/security scale without being questioned or stymied by lowly citizens or private companies. Even if he’s right about that (he isn’t), he’s wrong about the balance. This isn’t privacy vs. security. This is security vs. insecurity. For a speech so front-loaded with tales of security breaches and malicious hacking, the back end is nothing more than bad arguments for weakened encryption — something the government may benefit from, but will do nothing to protect people from malicious hackers or malicious governments.

All the complaints about a skewed balance are being presented by an entity that’s hardly a victim. Electronic devices — particularly cellphones — generate an enormous amount of data that’s not locked behind encryption. The government can — without a warrant — track your movements, either post-facto, or with some creative paperwork, in real time. Tons of other “smart” devices are generating a wealth of records only a third party and a subpoena away. And that’s just the things citizens own. This says nothing about the wealth of surveillance options already deployed by the government and those waiting in the wings for the next sell off of civil liberties

It also should be noted Rosenstein is trying to make “responsible encryption” a thing. He obviously wants the word “backdoor” erased from the debate. While it’s tempting to sympathize with Rosenstein’s desire to take a loaded word out of the encryption debate lexicon, the one he’s replacing it with is worse. As Rob Graham at Errata Security points out, the new term is loaded language itself, especially when attached to Rosenstein’s bullshit metric: “measuring success in prevented crimes and saved lives.”

I feel for Rosenstein, because the term “backdoor” does have a pejorative connotation, which can be considered unfair. But that’s like saying the word “murder” is a pejorative term for killing people, or “torture” is a pejorative term for torture. The bad connotation exists because we don’t like government surveillance. I mean, honestly calling this feature “government surveillance feature” is likewise pejorative, and likewise exactly what it is that we are talking about.

Then there’s the problem with Rosenstein deploying rhetorical dodges in his discussions about encryption, which presumably include a number of government officials. Alex Gaynor, who worked for the United States Digital Service and participated in the Obama Administration’s discussion of potential encryption backdoors, points out Rosenstein’s abuse of his position.

Mr. Rosenstein plainly wants to reopen the “going dark” debate that began under the previously administration, spearheaded by FBI Director Jim Comey. While I disagree vehemently with him, it’s a valid policy position – and I have every reason to believe him that there are investigations in which encryption does hamper the Justice Department and FBI’s ability to investigate. However, he is not entitled to mislead the public in order to make that point. And make no mistake. Attempting to use the spectre of familiar computer security challenges in order to make the argument that his policy is necessary, even though his policy has nothing to do with these challenges, is the height of intellectual dishonesty.

There’s an endgame to Rosenstein’s dishonest rhetoric. And it won’t be tech companies being guilted into participating in his “responsible encryption” charade. It will be backdoors. And they will be legislated.

The Deputy Attorney General says that he is interested in “frank discussion”. However, his actual remarks demonstrate he is interested in anything but — his goal is to secure legislation akin to CALEA for your cellphone, and he doesn’t care who he has to mislead to accomplish this. Mr. Deputy Attorney General, I expect better.

This is what the DOJ wants. But Rosenstein is too weak-willed to say it out loud. So he spouts this contradictory, misleading, wholly asinine garbage to whatever audience will have him. Rosenstein is obtuse enough to be dangerous. Fortunately, most legislators (so far) seem unwilling to sacrifice the security of citizens on the altar of lawful access.

Filed Under: backdoors, doj, going dark, james comey, nerd harder, responsible encryption, rod rosenstein