For a while now we’ve been mocking the Trump White House’s plans for an “AI framework” that would have the frontier AI labs hand over their top models for an initial review. After all, this was more or less the exact same plan that the Biden admin worked out in 2023, but it was done in a thoughtful and careful manner. And it caused a bunch of the VC bros in Silicon Valley to come out in support of fascism, while claiming it was a necessary defense against Biden’s attack on supposedly open innovation. Of course, all of that was bullshit, and that’s made even more clear by every step the Trump White House has taken to reinvent a similar “voluntary” AI review plan, but dumber.
The White House does not plan to publicly release its new framework for evaluating advanced AI models, three sources familiar with the discussions told Axios.
The Trump administration invited staffers from OpenAI, Anthropic, Google, Meta, Nvidia, and other leading AI companies to the White House on Tuesday to share an overview of its new AI oversight framework, the people said. AI developers will have the ability to voluntarily submit new models to the federal government up to 30 days ahead of their public release. The White House will then vet their cyber capabilities according to a classified benchmarking system and share the AI models with federal agencies and trusted corporate partners.
The White House isn’t sharing more information about its testing criteria or which AI models will be covered by the framework, though open models will reportedly be excluded, according to Axios. That has left smaller AI startups, safety advocates, and third-party researchers in the dark about crucial aspects of how the federal government is addressing the cyber risks posed by advanced AI systems. Some argue that the secretive process will give an advantage to larger companies.
Considering that the likes of Andreessen Horowitz (investors in OpenAI) claimed they had to support Donald Trump over Joe Biden because they would support anyone who agreed with their “little tech agenda,” I’m curious how they can possibly square that with the fact that this new framework is significantly worse than the Biden framework, specifically for the “little tech” companies that a16z has used as a shield to defend their support for authoritarian politics?
Of course, the other reason why the White House is probably keeping the framework a secret is because it would show how incompetent they are. All the reporting so far suggests the entire process has been a clusterfuck, which is much more about which companies get to set up which regulatory moats to protect their own business models, rather than what’s best for either innovation or the American public.
At Nvidia, Microsoft, Google and Meta, executives grew increasingly concerned that Anthropic and OpenAI would win over the White House with their arguments for tighter restrictions, according to two of the people. That would potentially cement the A.I. start-ups’ positions as market leaders,
Other A.I. labs were at risk of falling permanently behind, the people added. And because several of the companies make their own open-source models or supply hardware to businesses that use open-source technology, they worried the restrictions could harm them.
Over private texts, phone calls and video conferences, executives quietly built an argument that open-source models were good for the world and for American innovation, according to three of the people familiar with the talks.
But, of course, that’s just the way things work when you have a White House that makes decisions entirely based on transactional motives, rather than anything involving principles.
As we discussed last week, so much of this is all about whose vision of the AI world wins out — whether a handful of giant companies get to lock in the regulatory moat they’ve built for themselves, or an actually competitive market lets people make their own decisions and keep control over their own experiences. Maybe that’s the real reason nobody’s allowed to see the rulebook: because it would reveal who the administration agreed to let write the rules.
On an afternoon in mid-May, dozens of Microsoft engineers and their managers gathered online and in a conference room at the company’s Redmond, Washington, headquarters to discuss Project Glasswing.
The tech giant was racing to fix weaknesses in its code that a new AI model known as Mythos was uncovering at an unprecedented clip. The AI behemoth Anthropic, which developed Mythos, had given access to select organizations that make software used by regular people, companies and governments across the world. The goal was to find and fix the vulnerabilities before hackers and adversarial governments like China began using similar tools to find and exploit them for espionage and sabotage.
As the group settled in, one engineer asked the question that loomed over the meeting: Did Mythos “live up to the hype that Anthropic claimed it would have had?”
“Yes,” a manager responded, according to a recording of the meeting viewed by ProPublica.
The version being used by Microsoft, Claude Mythos Preview, was surfacing bugs faster than the tech giant could patch them, and engineers, the manager said, were now in “a mad dash” to close the gap.
One slide in that day’s presentation showed that in April alone, Mythos had uncovered 90 “critical” bugs and 141 “important” ones in SharePoint, Microsoft’s widely used collaboration software. In the first half of May it found even more.
“Please, please, please if your org has any April bugs, drive those down,” engineering manager Hans Andersen implored the group. They had roughly two weeks “to find as many things and do as much good as we can with this access.”
May 31, he explained, “is considered the day when the rest of the world will have caught up.”
The engineers on the call poked at that assertion, with one of them summing up the predicament: “So basically you’re saying if it’s released on June 1, then on June 2 the adversaries will have our bugs?”
Yep, one person responded. Yep, another echoed.
Ever since Anthropic kick-started a national conversation about the bug-hunting power of AI in April, when Project Glasswing was made public, national security experts predicted that the U.S. would have a window of opportunity to fix flaws before adversaries would have similar models capable of discovering the same weaknesses. In late June, the international alliance of intelligence agencies known as the Five Eyes — whose members are the U.S., Australia, Canada, New Zealand and the U.K. — warned in an unusual joint statement that in a matter of months, that window would be closing. But the recording of the Microsoft meeting, along with internal documents reviewed by ProPublica, suggest the day of cyber reckoning may already be here.
Given the deluge of flaws Mythos has identified, Microsoft so far has focused on patching those it considers most dangerous, which are classified critical or important, according to the presentation as well as the company’s own public patch updates. The internal records indicate that Microsoft plans to eventually address “moderate”-severity flaws uncovered by Mythos. The documents made no mention of “low”-severity bugs.
The company’s approach reflects the triage system that is typical in the industry. Just as the sickest patients are the first to be treated in the emergency room, vulnerability triage prioritizes issues that are likely to cause the most damage if exploited by hackers.
But that strategy carries its own risk in this AI-powered bug-finding era, in which new tools are unearthing a record-breaking volume of weaknesses in the products we use every day. Mythos, for example, is able to chain together a string of bugs that build on one another, meaning that the low- and moderate-severity vulnerabilities that remain unpatched could create an opening to carry out devastating attacks.
“The problem now is that you can chain four low-level flaws, and that can equal a high severity,” said Vinh Nguyen, a senior technical adviser to Anthropic and a senior fellow for AI at the Council on Foreign Relations who formerly served as chief AI officer and chief data scientist at the National Security Agency. “If you’re Microsoft, the current triage strategy may be underpricing risks.”
In emailed responses to ProPublica’s questions, Microsoft stood by its approach, saying its triaging decisions are based on a number of factors, including exploitability and the impact on customers. The company presentation did not mention chaining, but a spokesperson told ProPublica that the technique “has long been considered as part of vulnerability assessment and risk analysis.”
Asked about the internal presentation and the then-looming May 31 deadline, the spokesperson downplayed its significance, saying that “accelerated targeting and exploitation of new vulnerabilities is not a new phenomenon.” That said, he added, the comments made during the meeting reflect how the company “feels a sense of urgency to help our customers at this time.”
“What was heard on that call and is true today is that security is Microsoft’s most important priority and teams across the company are prioritizing using AI to discover and remediate vulnerabilities as quickly as possible.”
Microsoft declined to answer questions about how many bugs engineers had patched since the presentation.
Anthropic declined to comment.
The internal Microsoft presentation and accompanying slides predicted that the group of staffers working on SharePoint, which is used by governments and businesses worldwide to manage data and documents, “will be busy for months,” first working through the highest-priority critical bugs then tackling the important ones in August. Microsoft says vulnerabilities it categorizes as critical include so-called worms that can crash systems and spread malware as they race across computer networks. Important ones could result in “compromise of the confidentiality, integrity, or availability of user data” as well as the “availability of processing resources.” After those categories were cleared, the group would begin work on roughly 300 “moderate” bugs, according to the presentation.
While the internal documents reviewed by ProPublica do not include updates on the entire breadth of Microsoft’s offerings, they do give a sense of the scale of the problem. One document noted that, since the company started using Mythos earlier this year, it had collectively found hundreds of bugs that Microsoft categorized as either critical or important in popular products such as Microsoft 365, the Teams conferencing platform and the Copilot AI tool. As of mid-May, most of them had yet to be patched.
“They’re not profound and exotic, but they’re real,” Andersen, the engineering manager, said during the meeting. “And a lot of them are exploitable.”
It’s unclear whether hackers have exploited any specific bug identified by Mythos, but some have tapped AI to automate attacks and appear to be using Mythos-like tech to find and exploit weaknesses.
There have been outward signs of Microsoft’s internal struggle to deal with the growing list of bugs to be patched. Each month, the company publicly releases fixes for its software vulnerabilities in what’s known as “Patch Tuesday.” In June, it released patches for more than 200 bugs, which industry experts then said was an all-time high. But on July 14, the company blew through that record and released patches for more than 600 bugs. Only seven were categorized as low- or moderate-severity, one of which hackers were actively exploiting, according to Dustin Childs, leader of the Zero Day Initiative bug bounty program, which is part of cybersecurity company TrendAI. The rest were important or critical.
“Well folks. Here we are. The bug apocalypse has fully descended upon us,” Childs wrote in a blog post on July 14.
Microsoft told ProPublica that the overall volume of bugs “will not be plateauing for a bit,” but a spokesperson said the company has “invested heavily in both people as well as AI-powered triage solutions that scale quickly to handle the growing number of vulnerabilities.”
Given the new realities of the AI age, including the chaining capabilities, companies like Microsoft might need to rethink their entire approach to triage, said Nguyen, the NSA’s former AI chief. Rather than shunting what are now considered low-risk flaws aside, companies should be dedicating staff to developing and testing patches for the entire spectrum of vulnerabilities, he said. In other words, the cyber ER needs more doctors and nurses treating illnesses that are life-threatening as well as the minor wounds that could later turn deadly.
“There’s no alternative,” Nguyen said. “The patients are coming in fast and furious.”
Microsoft told ProPublica it’s “always going to be reevaluating and considering whether things that were previously lows or moderates be upgraded or thought about differently. With these AI systems, it makes us rethink some of these things. Across the industry, we’re all looking to see how drastic of a change it will be.”
Microsoft’s users may be particularly vulnerable. The popularity of its offerings, used the world over, makes it a frequent and lucrative target for hackers. In addition, many of its products contain “legacy” code. Developed decades ago using now-outdated technology, this code contains unaddressed flaws and contributes to what is known in the industry as “technical debt.”
But the challenge of fixing the flood of newly found bugs also extends to the rest of the software industry, and to open-source software code that is typically free to use and largely maintained by volunteers. Open-source software underpins internet infrastructure and is incorporated into much of the world’s modern technology, including products offered by major tech companies such as Microsoft.
“Nobody has really figured out how to deal with this, and everybody is casting around for what they need to do,” said J. Michael Daniel, a former cybersecurity adviser to President Barack Obama and the president of the Cyber Threat Alliance, a nonprofit organization focused on cybersecurity. “Our tech debt is coming due.”
Ben Edwards, a data scientist who specializes in managing software vulnerabilities, said the software industry was handling an “intense volume even before AI.”
“It was like drinking from a garden hose on the jet setting before, and now it’s like drinking from a fire hose,” Edwards said. “They might have had the teams that could handle that garden hose. Whether they can handle the fire hose is something else.”
Although the volume of vulnerabilities has grown over the years, Microsoft’s internal group responsible for fielding them, the Microsoft Security Response Center, has been perennially understaffed. Even before the crush of AI-identified bugs, the center fielded hundreds or even thousands of reports a month, pushing the group to its limits, ProPublica has reported.
The size of the center reflects Microsoft’s corporate philosophy: Plugging security holes is a cost center, while making new products is a profit center, former employees said. The company is loath to tie up its best engineers with making security patches — a cost center — instead of developing new products and features that will generate profits, ProPublica has reported.
Microsoft told ProPublica that it does not discuss internal staffing decisions but has made investments in recent years to “focus our teams on keeping our customers secure.” The company “continuously evaluates the staffing, processes, and technologies required to support security response and vulnerability management,” a spokesperson said.
According to the slides that accompanied the May internal presentation, Anthropic provided Mythos access to roughly 50 full-time Microsoft employees, with a goal to “harden critical services before publicly available models catch up.” A slide titled “What’s Next” predicted that the Microsoft Security Response Center would see continued case volume “as public tools catch up” to Mythos.
During the May meeting, one staffer appeared to take comfort in the belief that adversaries “don’t have the source code” that such an AI tool would scan for weaknesses. His colleagues, however, quickly corrected him. Portions of Microsoft’s code have, in fact, fallen into hackers’ hands over the years.
“It might not be this week’s source code,” one person said. “But they’ve got source code. It’s out there.”
In a statement to ProPublica, Microsoft downplayed the comment, saying engineers “design our security processes on the expectation that determined adversaries may gain access to code.”
Ideally, I think it makes sense to open up the U.S. market to Chinese goods (AI, EVs, robots), but have functional and well-funded regulators that policy all labor, competition, environmental, privacy, and consumer rights abuses. But big companies don’t much like that because it would boost competition and accountability, eroding quarterly revenues.
So instead we get protectionism. Or in the case of the Trump administration, protectionism run by a bunch of corrupt, incompetent clowns who gutted most of our federal regulators, then saddled them with an elaborate new policy ask that’s impossible to implement, even if the admin were competent.
Which it isn’t.
If you want a good idea of what Trump protectionism looks like in practice, I recommend this Verge piece about the Trump effort to ban Chinese drones. Spoiler: it’s not going well, and it’s trivial to trick the administration and bypass the ban without doing much, resulting in U.S. consumers losing access to lower-cost popular goods (DJI was the most popular and successful drone maker by far) under the pretense this is helping Americans and rekindling U.S. domestic electronics manufacturing.
Having unsuccessfully “blocked” Chinese drones (read: caused a bunch of annoying and pointless chaos), the Trump administration is dramatically expanding its protectionist campaign to include everything from cheap Chinese power converters (necessary for the world’s migration away from fossil fuels), to low-cost Chinese vacuum robots:
“While the government did include “humanoid robots” and “quadrupeds” among the bots it wants to ban, the ban is far broader than that. It covers almost any new software-controlled robot that travels over the ground, weighs more than 4.4 pounds (including any dock), can perceive its environment, and has wireless connectivity.
By that definition, the FCC is banning future robot lawnmowers, sidewalk delivery robots, and the robots that crate around packages at your local Amazon warehouse, too.”
Great stuff. Surely this will be competently enforced by… (checks notes)… Brendan fucking Carr.
Like his bumbling and unpopular boss, Brendan Carr has always made a gigantic stink about China. If you recall, he was one of the biggest proponents of a “ban on TikTok,” which he claimed was necessary to protect national security and stop propaganda. The solution to that problem was to steal the company and offload it to Trump’s billionaire friends so they could spread propaganda and abuse privacy.
The stupidity, xenophobia, and corruption on display with the Trump “TikTok ban” is going to be adapted for AI, and the next six to twelve months of U.S. tech policy will be dumber and more chaotic than ever. Especially as the biggest U.S. AI companies — swimming in debt, facing a data center glut bubble, and a long way from profitability — look to the government to protect them from all manner of competition (foreign, on device, open source, whatever).
This may come as a surprise to you, but Trump incorporated doesn’t actually care about protecting U.S. consumers from Roomba privacy abuses, propaganda, or national security abuses. If they did, they wouldn’t be supporting Trumpism. If they did, they’d regulate data brokers and pass a modern internet privacy law that applied to every company and executive doing business in the U.S.
Being mindlessly and personally transactional, the Trump administration is also keen to erect additional troll and tribute systems if companies want to continue doing business in the U.S. The money will go to his business interests and allies, but in a bid to appeal to nationalists, it’s going to all be layered under the pretense that this is all going to magically shift robotic manufacturing back to the U.S:
“But again, the government isn’t asking these companies any questions about security — not one — to get the waiver that lets them through. The FCC only wants to know where they’re designed and made and assembled and tested and influenced, and get a specific commitment to start manufacturing them in the US instead.”
But they’re not going to be competent enough to make any of this happen, even if shifting the entirety of low-cost electronics manufacturing back to the U.S. was a thing you could actually accomplish. A key part of the problem is that, as we saw from Trump’s dream of having U.S.-made smartphones, the folks in charge of this effort have a child-like understanding of how everything works. And everybody else — like massive swaths of the business community, academia, and the press — are too feckless to stand up to the corrupt stupidity in any meaningful way.
The Trump administration has also gone out of its way to ensure our federal regulators no longer have the resources or legal autonomy to function. Then they’re saddling them with this massive new layer of protectionism they’d be incapable of enforcing even if they wanted to. They’re not even bright enough to understand that even base protectionism is out of their reach.
This is just racist, crony capitalism by corrupt and incompetent zealots, and it’s going to cause irreparable harm to consumer prices, product quality, product availability, hobbies, science, the shift to renewables, and everything else over the next two to three years. All propped up by a lazy press, self-serving domestic monopolists, and a whole lot of bullshit about privacy and national security by the kind of people who are a direct and obvious threat to both.
On a laptop screen in a dimly lit tent near the Donetsk front, a Ukrainian drone team steers its aircraft into the turret of a Russian T-72 and glimpses the start of an explosion before the picture dissolves into static. The strike is uploaded, verified, and scored against the point value assigned to the tank. The unit climbs a public leaderboard that ranks hundreds of drone teams, and the points are currency: a higher score buys better equipment, faster, from an online marketplace the warfighters compare to Amazon.
Washington is about to decide how the federal government will parcel out access to the most powerful AI, and it is drifting toward concentrating that capability in a few chosen hands, rationed by criteria no one outside the process can see. A country with foreign invaders on its own soil has spent the past year learning to do the reverse—and winning back ground as it does. From Luhansk to Lviv, Ukraine puts its best tools in the hands of whoever can use them and shares what it knows about the enemy as fast as it safely can, openly and by rule.
Behind the leaderboard sits a set of arrangements Ukraine built under fire. A marketplace lets frontline units order drones directly from hundreds of manufacturers, most of them small shops scattered across Ukraine. A procurement cycle that once ran months now takes days, and new designs reach the trenches within about a month of leaving the workbench, because the units doing the fighting, not a distant acquisition office, decide what they need in the field. Furthermore, their feedback goes straight back to the manufacturer, sometimes the same day. Because the manufacturing is dispersed rather than massed, no single Russian strike could ever change much.
Ukraine has been just as willing to share what it learns. Late last month its defense ministry opened a platform called TrophyLab that hands the technical anatomy of captured Russian weapons—schematics, known vulnerabilities, even physical samples—to a deliberately wide circle: allied militaries and intelligence services, and hundreds of Ukrainian and partner-country firms. Access is vetted and revocable, governed by published criteria. The premise is that knowledge of a threat is worth more shared than hoarded. This should be a rule everyone can see, rather than the whims of a distant official.
That combination, wide but rule-bound, is exactly what the executive order the White House issued in June fails to deliver. Faced with AI systems that can now find software flaws faster than any human team, the order promises early access to the most capable models to a few “trusted partners”—a phrase it never defines, routed through a classified process. It calls the arrangement voluntary. In practice it has not been: under national-security and commerce authorities the administration has already restricted, suspended, and then cleared frontier models, with no published criteria anyone outside the process can point to. Ukraine’s leaderboard may be a crude way to run a war, but it is at least a rule—public, legible, the same for every unit.
The deeper problem is what that opacity does. Ukraine found that capability does the most good spread widely, that a defense holds because it has no single point of failure, and that threat intelligence should travel by rule rather than favor. A trusted-partner tier governed by undefined discretion inverts all three: it concentrates the best tools among those already best equipped, builds the very chokepoint Ukraine works to avoid, and turns shared knowledge into something rationed by judgment no one can inspect. The United States already runs sector-based centers for sharing threat intelligence; the question the framework raises is not whether to centralize but whether the flow reaches the defenders who need it or stops at a favored few.
And there are real lessons for the United States. Ukraine’s openness may look like the underdog’s strategy, and the United States is the wealthiest, most powerful country on earth—but national strength does not mean every system is strong. The defenders who most need help are not the money-center banks and wealthy university hospitals; they are smaller institutions that, despite non-specific promises they’ll be helped, seem unlikely to benefit from this system as it’s set up. For them, a head start reserved for the already-strong is no help at all.
Ukraine did not arrive at any of this by design. It was forced into it, and used a mix of openness and clear rules to stop a much larger power in its tracks. Washington has the luxury of choosing on purpose but may be drifting towards a system governed by whims and favoritism rather than clear rules and standards.
Eli Lehrer is president and co-founder of the R Street Institute.
It’s not that AI can’t be useful or helpful in certain contexts. And this is certainly not to say AI can’t take over repetitive tasks to allow people to focus on things that need more of human touch.
The problem with AI isn’t necessarily AI itself. It’s that far too many tech companies are pitching AI as a one-size-fits-all solution to pretty much everything. And far too many entities are taking tech companies at their word, with disastrous results.
Axon (formerly Taser) has cornered the body cam market and is now trying to sweep up everything else. While it still hasn’t made a foray into facial recognition tech, it’s pitching products (Draft One and Form One) that use AI to automate report writing for police officers. Draft One has been pitched as a time-saver — one capable of transcribing body cam audio to generate police reports. The company’s CEO Rick Smith thinks this add-on to its body cam products with “potentially free up 25% of an officer’s time.”
Considering a lot of officers spend most of their time engaged in pretextual stops, this isn’t really good news. It just means officers will be able to violate rights more frequently with no perceivable benefit to public safety.
An artificial intelligence that writes police reports had some explaining to do earlier this month after it claimed a Heber City officer had shape-shifted into a frog.
However, the truth behind that so-called magical transformation is simple.
“The body cam software and the AI report writing software picked up on the movie that was playing in the background, which happened to be ‘The Princess and the Frog,'” Sgt. Keel told FOX 13 News. “That’s when we learned the importance of correcting these AI-generated reports.”
Weirdly, this anecdote comes from the same law enforcement agency that claims AI-generated police reports keep this Utah city “safer.” The report doesn’t explain how this is being accomplished. Sgt. Keel simply says the tech saves him “6-8 hours a week.” Sgt. Keel does not explain what’s being done with these extra hours.
The other problem with relying on AI to generate police reports is that officers are generating a new layer of plausible deniability. If errors are found, cops can blame it on the algorithm. Beyond that, there are problems cops pretend don’t exist, like making any testimony reliant on AI-generated reports instantly suspect. If cops aren’t writing their own reports, they can’t possibly claim these statements are their own under oath.
But even if you ignore all of that and choose to focus on the things companies like Axon would prefer you to direct your focus to, we’re still not seeing the sort of improvement that would theoretically offset the downsides of relying on AI. Trial runs of Axon’s new Form One AI product haven’t exactly been a resounding success, as Thomas Brewster reports for Forbes.
[A]fter seven months of testing the technology, many of Lafayette’s officers found the tool was wasting time, not saving it. “I know it doesn’t save me time and I know it has inaccuracies that I will have to edit,” wrote one officer in a cache of emails obtained by Forbes via public records request. Form One struggled to record the right names or car plates, even when they were clearly stated in the camera footage, according to other emails. One cop said a simple form that used to take 30 seconds to fill out manually now takes three minutes with Form One because there are so many errors. “Form One dramatically increases the time it takes to finish reports,” he wrote to a colleague.
That’s just the experience of a single town of 70,000 people in Indiana. Imagine having this amount of routine failure applied to departments with hundreds of officers and thousands of daily reports. What’s worse is that Form One is far less sophisticated than Draft One, which is used to transcribe body cam footage. The software is only expected to accurately add names and addresses to relevant sections of police reports. If it can’t be trusted to do this, there’s little reason to believe Draft One can provide an accurate accounting of a police stop by transcribing audio.
Axon claims these are not indicative of whatever the final product will be. According to the Axon spokesperson, Lafayette was granted “early access” to an (apparently) unfinished AI tool. Axon implies the final product will be better, but neither the Lafayette PD or Axon itself were willing to provide any info that might allow critics to move this from implication to inference.
Manchester, New Hampshire’s police department had roughly the same experience with Draft One.
Ian Adams, a former police officer and criminology professor at the University of South Carolina, studied the Manchester Police Department in New Hampshire’s use of Draft One. Adams analyzed time stamps for when officers started a report and when they filed it. Only some had access to the software. The study found that there was no improvement in how long it took for cops to file reports because they spent a significant amount of time editing the AI system’s work: removing irrelevant information, fixing inaccuracies and adding important facts it missed. “It was just easier to type the report themselves,” says Manchester’s Lieutenant Matthew Barter, who participated in the research.
Weirdly, most officers still thought Draft One sped up report writing, despite data showing otherwise. And even if officers were convinced Draft One was more efficient that simply writing reports themselves, the PD apparently decided the data was more accurate than a bunch of subjective opinions. And it wasn’t the only beta tester (so to speak) to do so:
Even though it was the first agency to test Axon’s Draft One, Manchester ditched it in 2024. The same year, the Anchorage Police Department in Alaska also decided to stop using it, citing zero time savings.
Presumably, these agencies weren’t required to pay for this subpar tech. And while it’s safe to say the tech will continue to improve, the question is whether it will ever be worth it. With fewer courts willing to accept AI-generated court filings and becoming increasingly skeptical of the tech in this context, cop shops are going to be paying for a product that generates reports they can’t submit as testimony or evidence.
Beyond that, the tech apparently needs so much human backstopping that the job of writing reports may as well just be handed back to the humans. Even if it does improve to the point that it can actually reduce the paperwork load on officers, no police department enthused about the tech has specified what will be done with all of this new free time. If it’s just going to be more of the same old policing, the public gains nothing but additional chances to have their rights violated. If the free time is going to be used to build community relationships and route more officers to investigations that might contribute to overall public safety, some sort of trade-off might be acceptable. But from what’s been demonstrated so far, AI is just compounding errors without providing any real usefulness to the communities these law enforcement agencies serve.
There’s been a bunch of news this week regarding Minnesota’s new law that purports to prohibit “nudification” technology, and the fact that xAI has sued to have the law blocked as unconstitutional. A few things need to be said upfront, because it’s very, very easy to just say the tech is terrible, that Elon Musk and Grok are terrible, and that of course Minnesota should ban it. But it’s also possible that, in the rush to attack very problematic apps built by very problematic people, Minnesota drafted a bad law that is ridiculously overbroad and pretty clearly unconstitutional. And… that is exactly what appears to be the case.
Let’s start with the basics: apps (mostly powered by various AI tools) that are used to produce modified imagery, especially stripping people of their clothes are… bad. They should be socially shunned. People using them to objectify or sexualize others are doing bad things, and people should judge those who use those apps accordingly. This is not a defense of those apps. Similarly, Elon Musk’s Grok and its widely promoted use of putting people (including children) in bathing suits definitely deserves social shunning as well. Norms take time to form, and the shunning here is still catching up to the technology.
But passing a badly drafted, obviously unconstitutional law does not help form those norms. Nor does it punish Elon Musk. Instead, it allows him to act like a First Amendment martyr.
It’s also worth clearing something up early, because a lot of the coverage has gotten it wrong: this is not a law about child sexual abuse material. CSAM is already quite illegal under both state and federal law, and nothing in HF 1606 is limited to images of minors. Had Minnesota drafted a law narrowly targeting AI-generated CSAM, it might have survived a constitutional challenge. That’s not what it did.
And if you want to pass a law to ban technology like this, there are rules under the First Amendment. And, in Minnesota, we even know what some of those rules are. After all, a decade ago, the state also passed a law criminalizing the dissemination of “nonconsensual private sexual images.” After some back and forth in the courts, the Minnesota Supreme Court finally blessed the law as constitutional in late 2020, but made it quite clear that the law went right up to the First Amendment line. It first noted that while the state wanted to claim there’s an entirely new category of unprotected speech (in this case, “substantial invasions of privacy”), the court refused to do so, citing the famed US v. Stevens case (about an attempt to outlaw animal “crush” videos) in which the Supreme Court made it quite clear that it wasn’t open to creating new categories of unprotected speech:
The United States Supreme Court has emphatically rejected “freewheeling” attempts “to declare new categories of speech outside the scope of the First Amendment.” Stevens, 559 U.S. at 472; see also Jorgenson, 946 N.W.2d at 604 (“The United States Supreme Court has been reluctant to expand these traditional categories of unprotected speech.”). It is possible, however, there are “some categories of speech that have been historically unprotected, but have not yet been specifically identified or discussed.” Stevens, 559 U.S. at 472.
To successfully argue for a new unprotected category of speech, the proponent must present “persuasive evidence that a novel restriction on content is part of a long (if heretofore unrecognized) tradition of proscription.” Brown v. Ent. Merchs. Ass’n, 564 U.S. 786, 792 (2011). This is a heavy burden to bear, and the Supreme Court has recently rejected creating new categories of unprotected speech for animal cruelty, Stevens, 559 U.S. at 472, depictions of excessive violence, Brown, 564 U.S. at 791–93, and false statements, Alvarez, 567 U.S. at 722–23.
In this case, we conclude that the State has failed to carry the heavy burden required to provide a basis to establish a new category of unprotected speech.
And yet, the law was still deemed constitutional, but not because it created a new category of unprotected speech, but rather because it passed strict scrutiny, in which the law is narrowly tailored to use “the least restrictive means” of addressing a compelling government interest. That is the test by which a law can still be deemed viable under the First Amendment, despite suppressing speech. In the case of the nonconsensual imagery bill, the law passed strict scrutiny because it focused very narrowly on a category of speech that is very likely to cause harm, and put in place a law that was narrowly tailored to only target that speech, and on top of that included clear exemptions for edge cases that likely wouldn’t be harmful.
Indeed, the court leaned hard on the fact that the law only reached images disseminated without consent, and only when the disseminator knew or reasonably should have known the subject expected privacy. Those two limits — consent and intent — are what kept the statute from sweeping in vast amounts of protected speech. Some quotes from the court which list out all the factors necessary to pass strict scrutiny.
First, the Legislature explicitly defined the type of image that is criminalized…. Furthermore, the image has to be “obtained or created under circumstances in which the actor knew or reasonably should have known the person depicted had a reasonable expectation of privacy.” Id., subd. 1(3). Images that do not clear each of these hurdles fall outside the scope of the statute.
Second, a defendant must “intentionally” disseminate the image. … This mens rea requirement means that a defendant must knowingly and voluntarily disseminate a private sexual image; negligent, accidental, or even reckless distributions are not proscribed. This specific intent requirement further narrows the statute and keeps it from “target[ing] broad categories of speech.”
Third, the statute has seven enumerated exemptions…. The statute allows for private sexual images to be distributed “in the course of seeking or receiving medical or mental health treatment.” Id., subd. 5(3). Advertisers, booksellers, and artists are protected because images “obtained in a commercial setting” for legal purposes fall outside the statute’s reach. Id., subd. 5(4). Journalists cannot be prosecuted because there are exemptions for the dissemination of private sexual images that involve matters of public interest and “exposure[s] in public.” Id., subd. 5(4)–(5).8 Educators and scientists are protected because there is an exemption for private sexual images disseminated for “legitimate scientific research or educational purposes.” Id., subd. 5(6). Accordingly, even if protected speech falls within the ambit of subdivision one and a disseminator acted with the requisite mens rea, that person may still be exempt from prosecution under these precise exceptions.
Fourth, to be prosecuted under the statute, a disseminator must act without consent…. This provision provides additional protection for commercial advertisements, certain adult films, artistic works, and other creative expression outside the statute’s scope.
Finally, this statute only encompasses private speech…. Unlike the overly broad statutes at issue in our recent decisions in In re Welfare of A.J.B. and Jorgenson, this statute covers only private sexual images and does not prohibit speech that is “at the core of protected First Amendment speech.”
It was all of that combined that allowed the law to pass strict scrutiny — something that is incredibly difficult to do. Most laws that have to clear strict scrutiny don’t. Here, this law survived with a careful roadmap from the court of how to do so.
One would think that Minnesota legislators would be aware of this ruling and the clear reasons why the law was deemed to pass strict scrutiny and then write an equivalent law with the same elements in trying to ban nudify apps.
But for reasons known only to the Minnesota legislators, they basically ignored every single one of those points.
Minnesota’s anti-nudification tech law is not limited to non-consensual content. This means, as legal commentator Kathryn Tewson noted, that if she uploaded a picture of herself and asked Grok to put her in a bikini, she could by her own hand, cause Grok to break this law. That… seems like a very problematic law.
See, as I read this law, if I uploaded a picture of myself in a sundress and said “Grok, make this a picture of me in a bikini instead,” it would be a violation of the law for Grok to do that. I don’t think that should be illegal.
And, again, the Minnesota Supreme Court has already told the state pretty much exactly how to make this law constitutional: focus on nonconsensual imagery, narrowly tailor it to just the deeply harmful content, include an intent requirement, and include clear delineated exemptions for things that should be allowed.
Minnesota legislators did none of that. Indeed, even the definition of “intimate parts” in the law borrows its definition of ‘intimate parts’ from an earlier statute, covering: “the primary genital area, groin, inner thigh, buttocks, or breast of a human being” — not much of which is inherently sexual, let alone harmful. Tewson offers another example: an edit of a Taylor Swift photo that changes the texture of her fishnet stockings to look more like skin. Whatever tool made that edit just violated Minnesota law.
2. An image generated from this image of Taylor Swift performing on the Eras tour in which the texture of her legs in the modified image appears more similar to actual skin than it does to nude-fishnets-over-nude-tights:
This is, by definition, an overly broad, non-narrowly tailored law.
Another example: last year the TV show South Park did a deepfake parody of Donald Trump, showing a photorealistic version of him wandering naked through the desert, including his “intimate parts.”
Under this law, that video could violate HF 1606. That’s not narrowly tailored. That’s not dealing with intent or focused just on truly harmful content.
One lawyer I spoke to, after reading through the statute, wondered out loud whether the Minnesota legislature had deliberately drafted it in the dumbest way possible just to guarantee a successful challenge. That’s how poorly the law was drafted.
Of course, no one wants to hear that the law is badly drafted. Lots of people want to ban nudify apps and to yell about how ridiculous it is that Elon Musk has gone to court to challenge this law.
But… it’s the sort of thing he should be doing. Otherwise anyone can have Grok put themselves in a bikini and… Minnesota’s Attorney General can demand $500,000 for each such image created, even when the image was created deliberately, by the person in it, of themselves.
xAI (now a division of SpaceX) is right to challenge the law, not because nudify apps are a good thing, but because the law is terribly drafted and pretty clearly exceeds what’s allowed under the First Amendment. The complaint itself is worth a read. For one thing, it explains why xAI last week sued one of its own users for producing CSAM with Grok (which I had found perplexing at the time). It reads a lot like the company wanted a concrete example to put in this filing of how it fights back against those who use Grok in such ways (leaving out, of course, that Elon himself used the app to put himself in a bikini, thereby encouraging others to do the same).
It also explains why that complaint was focused on triggering the indemnity clause in X’s terms of service, which makes the user liable for any legal costs associated with their use of the product. What Musk is really signalling with that lawsuit is if Minnesota’s AG sues us under this law for your usage of the product, we’re going to sue you to cover our costs (which could include the $500,000 fine for any images created).
As the lawsuit notes, the law is just terribly written:
HF 1606 punishes AI platforms that allow users to alter images of real people to depict an “intimate part.” But the statute contains no knowledge, intent, or purpose requirement. It is a strict-liability statute keyed solely to whether a user succeeded in creating a covered image using the AI provider’s platform—regardless of whether the provider prohibits users from using its tool for such a purpose, regardless of how many mitigations the provider has in place, and regardless of how diligently the provider polices such conduct using its tool. There is no safe harbor for good-faith efforts of the provider of general-purpose AI creative tools to avoid harms. Liability attaches even if the depicted persons consented—or created the image themselves—and even if the image is never shared. Liability also attaches even if the image has artistic, scientific, political, satirical, educational, medical, or religious value, and (again) even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images.
Additionally, the law’s definition of “intimate part” is exceptionally broad. Although the federal government and various states have enacted statutes that clearly define nudity for the specific context of AI-generated images, Minnesota rejected such a precise definition. Instead, it borrowed the definition of “intimate part” from a criminal sexual-contact statute. That definition was drafted for nonconsensual touching and thus covers the inner thigh, buttocks, or breast of a man or woman, as well as the groin and primary genital area. HF 1606 accordingly bans ordinary depictions of men without shirts, people in shorts or swimsuits, and other body parts routinely displayed in public—far beyond what an ordinary person would consider “nudification.”
Even worse, as the lawsuit states, the bill’s “principal sponsor” admitted that the law was designed to apply to consensual imagery:
A service used by an adult to edit a photograph of him or herself or a consenting individual is covered on the same terms as a service used to create an image of an unwilling stranger. The statute’s text draws no distinction among them. And this was by design. When a staff member of the Senate Judiciary and Public Safety Committee pointed out that the Act’s “prohibition applies to consensual images,” Senator Maye Quade (the bill’s principal sponsor) explained “that is intentional.”
That is the bill’s main sponsor stating, on the record, that she deliberately chose to leave out one of the very features Minnesota’s own Supreme Court had identified as necessary for a law like this to survive constitutional scrutiny.
That is legislative malpractice.
Since the lawsuit was filed, Maye Quade and other legislators have publicly defended the bill:
“I don’t see this as a free speech issue. This does not regulate content; it does not regulate art. It regulates conduct,” Maye Quade said. “Prompts are not art, and we protect art specifically in this law. It’s pretty audacious to sue to prevent a law that protects children from being turned into child sexual abuse material.”
She’s describing a law she could have written, but didn’t.
Notice what’s missing from that defense: any explanation of why the consent and intent elements the Minnesota Supreme Court specifically identified as saving the 2016 law were left out of this one. Also, she’s just simply incorrect that the law does not regulate speech. Again, if she simply read what the Minnesota Supreme Court said about the nonconsensual intimate imagery law, it spent pages analyzing the nonconsensual imagery statute — a law covering narrower material than this one — as a content-based restriction on speech that had to pass strict scrutiny to survive.
Similarly, the law does not actually “protect art.” Its one and only exemption is if the work “requires the technical skill of a user to nudify an image or video.” That could protect some art, but not all. And it defines art only in a case where a level of skill is needed, which itself potentially creates First Amendment issues in defining what is, and what is not art. There is plenty of modern art that people regularly complain takes no “technical skill” to create.
The complaint itself includes some other examples of what would violate the law, including this (gross) AI-generated image that Trump posted of a slimmed down version of himself, some of his cabinet members, and a randomly generated woman in a bikini sitting in a gleaming blue reflecting pool. Under the law, whatever tool was used to generate that image pretty clearly violated Minnesota’s law:
In this viral snapshot—which President Trump posted publicly— President Trump, Vice President J.D. Vance, Secretary of State Marco Rubio, and Secretary of the Interior Doug Burgum all are portrayed shirtless in the Washington Mall’s reflecting pool, along with an unknown (possibly fictitious) woman.19 An “intimate part” (the breast) of at least the President, Vice President, Secretary of the Interior, and the woman are “depict[ed],” with the Secretary of State also at least arguably included as well. The President posted this image on his personal account, presumably to make light of the public controversy surrounding repairs to the reflecting pool on the National Mall.
Nudify apps are gross. Musk’s encouragement of people to use Grok to de-clothe people is gross. People who use AI tools to “nudify” people are gross. But that doesn’t mean all laws targeting such things are good laws or constitutional.
In this case, despite having clear instructions from its own Supreme Court on how to write a constitutional law, Minnesota’s legislature deliberately chose to write an unconstitutional one. And thus, this lawsuit is the proper thing for SpaceX/xAI/Musk to do.
Supporting the lawsuit is not supporting Elon or Grok or nudify apps. It’s telling every legislature in the country the same thing: if you want the law to survive, learn to draft it in ways that aren’t unconstitutional.
Just recently Karl warned that we were going to see some absolute nonsense as the US sought to somehow “ban” Chinese AI models from being used in the US. That seems to already be happening. It kicked off with talk that the US might “fight Chinese AI” using nearly identical arguments to what was used to ban (or force the sale of) TikTok before it. Some combination of “national security threat” combined with “oh no China” propaganda.
But most of the AI industry is now speaking out, in an open letter put together by Nvidia, against the potential path that the Trump administration considered taking: an attempt to ban or limit so-called “open weight” models. The companies seem to recognize that focusing on holding back these competitive models would actually do much more damage to the wider AI ecosystem.
There were notable exceptions from the campaign in defense of open weight models: Anthropic, OpenAI, and Google (the three leading frontier model labs) were not initially signed onto the letter. Though their absence quickly became the story — leading OpenAI and Google to reconsider and sign onto the letter days after it came out.
That left one major player off the letter: Anthropic (a company that has so far refused to release any open weight models). And now the company is trying to explain itself, but seems to only be digging itself a deeper hole.
One of the problems here is that the leading Chinese AI models tend to be open weight models, which can be downloaded and run locally, as compared to the leading frontier models from US companies which require you to access them via their own hosted models. Yes, most of the leading Chinese models also offer (sometimes significantly cheaper) cloud/API access to their models, but you can also run them yourself (for the smaller models directly on your own computers, or for the larger models via your own cloud setup).
There’s no inherent reason why the best open weight models are coming out of China, other than that they seem to have recognized that it may be the best way to get more people to use them and to compete against the American frontier models, which are much more proprietary and locked up. The strategy is a recognition that offering a compelling, more open alternative is how to get people to adopt your system over the American frontier models. If a generation of developers builds on top of Kimi or Qwen or one of the other Chinese open weight models, they become the de facto infrastructure for the next generation of digital tools.
In the same manner that Linux quietly became the substrate of the open internet, and it’s likely that an open weight model may become the equivalent for the next generation. Organizations may rely on frontier models for really deep work, but so much can be done with open weight models that a winner here becomes the commodity infrastructure provider for a new generation of software. That’s why any proposed restrictions on open weight models would get everything precisely backwards. It would guarantee that the wider open ecosystem gets built on non-American tools. Yet, the discussion around such bans seems to treat these as just another software product, rather than a fight over how the infrastructure of the internet will work going forward.
Of course, that’s not the only argument the Trump admin is using to try to stop these models. Last week they focused on claims that Kimi’s K3 model (the latest model to shake up the US market, despite being not quite as good as the frontier models) must have been “distilled” from Anthropic’s Fable 5.
“If we see, especially that overseas models are stealing from our great companies, we have the ability to sanction them because of this theft,” Bessent told Fox Business’ “Mornings with Maria” on Tuesday.
Bessent said the technical term for this theft is called distillation, which is an AI training method where a smaller, less capable model is built using outputs from an existing, stronger model. Anthropic sent a letter to the U.S. Senate Committee on Banking, Housing, and Urban Affairs last month alleging that the Chinese tech company Alibaba had carried out the “the largest known distillation attack” against it to date.
This is rich for a variety of reasons, not the least of which is that all of the frontier AI models were built by feeding their training models whatever information they could get their hands on, including (in Anthropic’s case) building a pirate library of downloaded books for which it had to pay out a pretty massive settlement to authors.
Distillation is not quite the same thing, but is functionally similar. It’s taking the work of an existing model to fine tune the model you’re working on. The claims about Kimi K3 seem somewhat exaggerated, as the initial claims were that it was distilled based on Anthropic’s Fable 5 release, but multiple people I’ve spoken to don’t see how that’s possible, given how Fable 5 has only been out for a little while (and then was turned off for a while due to the US government freaking out over nothing).
No matter what, distilled models are likely to be less powerful, and at least a decent period behind the frontier models, given that they’ll need access to the frontier models and time to train based on them. There’s also some dispute over how the open weight models may be using distillation, and which part of the training process works best.
But either way, the freakout over distillation seems… ridiculous. Bessent calling it “theft” is nonsense. Just as training a model on copyrighted works is a form of reading (which shouldn’t implicate copyright in the first place), so too is distilling, which is (in effect) training your model by having it compare its initial answers to similar answers from a frontier model and then adjusting based on the different results. It’s a form of learning based on observed results by others, not “stealing.” Pretending that it’s stealing or somehow should face sanctions or other consequences will put US AI development in a bad, bad spot.
Which brings us back to that letter. Here’s the case it actually makes:
Open weights also strengthen competition and competition is what keeps the gains of AI broadly shared rather than concentrated in a few hands. By allowing many organizations to build, adapt, and deploy advanced models, open weights create rivalry not only among model developers but across cloud chips, applications, and services. That competition spurs innovation, drives down costs, and distributes the benefits of AI broadly across our economy.
Open weights also give customers greater control. As organizations invest in AI, they want to know that they will not become locked into a single provider or lose the knowledge and capabilities they build over time. Open weight models help provide that assurance by allowing organizations to control their own data, evaluate and adapt models to their own needs, and deploy them wherever their business requirements demand. And as organizations create value with AI, open weights allow them to own that value through self-improving models, specialized capabilities, and accumulated knowledge that drive American sovereignty and prosperity.
Of course, Anthropic (which hasn’t released any open weight models) was conspicuously absent from the signatory block of that letter. Earlier this week, Anthropic’s Dario Amodei came out and tried to explain/justify the company’s stance, which boils down to: “we don’t think anyone should ban open weight models… but we do think the US should ban all the conditions that make quality open weight models possible.”
Amodei argues that simply banning Chinese open weight models wouldn’t solve the alleged “threats” that people are concerned about, though he admits directly that it would act as protectionist industrial policy that could benefit American AI companies (like Anthropic):
But banning the use of these models by US businesses does nothing to address this risk, because bad actors are unlikely to be legitimate US businesses. It would protect US AI companies from competition, but that has never been my goal.
It feels a bit like he’s protesting too much regarding the protectionism here, while trying to have it both ways. He claims he really has the best interests of safety at hand, and is against protectionist ideas, but it’s hard to square that with the rest of the article.
While he says the US shouldn’t ban open weight models (and it shouldn’t), he then puts a bunch of conditions on it, which would make it that much more difficult for the current crop of open weight models to compete. Namely, he leans in on the Sinophobia that has become popular these days in warning about “CCP” influence over models (which… should be less of a concern with open weight models, since those who use versions not hosted by the Chinese companies can adjust the models to deal with those concerns).
But then he says that we should punish Chinese AI companies for engaging in distillation:
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier. It is true that many of the companies carrying out these operations release open-weights models—but the open weights are far less relevant than the fact that the operations are backed by an authoritarian state seeking to overtake the US at the frontier. We should have policy interventions to deter this behavior. A blanket ban on open-weights models is neither the correct remedy nor something we have called for.
To be fair to Amodei, not everything on his list is competitor-hobbling. He also wants chip export controls tightened (a policy that predates this fight and has its own problems, but at least isn’t aimed at a business model), and he wants mandatory pre-release safety testing for all sufficiently capable models — open or closed, foreign or domestic, Claude included. That last one is the tell, though, and not in the way he intends: if you genuinely believe capability-based testing is the right lever, and you’ve just said bans “would protect US AI companies from competition, but that has never been my goal,” then what is the argument about distillation doing on the list at all? Testing catches dangerous capabilities regardless of how the model got them. The distillation crackdown adds nothing on safety. It only serves to kneecap cheaper competition.
And even the “safety testing” plank isn’t as neutral as it sounds. While safety testing is obviously important, when legally mandated, it can quickly turn into an expensive compliance-function of box-checking that only the largest companies can do, taking us back to the world of just a few providers, and limiting smaller competitive models from really being viable. While there are legitimate reasons for it, it can also create its own moat.
The proposed crackdown on distillation is just asking the state to step in and block lower-cost competitors from competing. Yes, these models can be competitive, but they should be driving the leading frontier models to continue to improve and to provide more value. What Amodei is asking for here is basically the US government to help prevent lower cost, lower quality competitors from pushing the floor of the AI market upwards.
Now, to be clear, as with any technology, you can claim that a more open, more widely available, more powerful version can be misused. But that has always been the case and we, in the US, have tended to default to allowing the technology to proceed, and figuring out ways to minimize the dangers/increase the good uses, rather than resorting to assuming the tech will be abused and working backwards to block all possible abuses. Historically, seeking to pre-vet technologies tends not to work well, and (often) opens up the market to foreign competitors to simply build better products.
The open letter makes a sharper version of this point, and you can see why Anthropic wouldn’t want to put its name to this point in particular:
Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk. It results in a small number of single points of failure, weakens competition, and leaves critical technology in the hands of a few providers. Open weight models, on the other hand, allow a broad community of researchers and developers to examine their behavior, identify vulnerabilities, develop safeguards, and improve them over time. Just as open-source software demonstrated that transparency can be more secure than obscurity, AI safety may depend on giving more people the ability to test and strengthen the models on which society relies. It allows for rigorous benchmarking and evaluation, red teaming, and protections tied to real and demonstrated harms rather than assuming that closed systems are safer by default.
Amodei also claims he supports the general argument of the open letter, but he disagrees with the idea that open weight models lead to better security:
This brings me to the open letter. I agree with much of it: open weights expand access to the AI economy, they strengthen competition at least for some use cases, and they give customers greater control. Concerns about distillation should be addressed through targeted legal and commercial frameworks—the same measure I described above. But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true.
This strikes me as a repeat of the age-old fight that always shows up in discussions of open source technologies: the claim that by making them open, security vulnerabilities are easier to find. Of course, what we’ve seen historically in other spaces is that this actually means that security vulnerabilities are more quickly patched, rather than in the “security by obscurity” space, where they can remain open (and possibly exploited) for much longer.
Amodei is asserting that the AI space is somehow different, though without much evidence for that other than what feels like a bit of fear-mongering about “weaponizing pandemic-level viruses.”
Of course, part of the problem here is that it often feels like Anthropic treats “crying wolf” as a marketing strategy, whereby much of the company is focused on talking up “our tools are soooooooo dangerous that you need us in there to protect you from them.” Even if there’s some truth to it, it’s awfully convenient that the same argument also happens to justify banning, punishing, or limiting the cheaper, more open, more user-controllable alternatives.
In the end, the federal government still might try to punish the Chinese open models in some form or another just because they view current American industrial policy in very nationalistic terms. But that won’t be good for the wider ecosystem, or for the general incentives to innovate. And, worst of all, it makes it that much harder to build a world where we’re not entirely dependent on a few giant companies controlling the “brains” of the tools the rest of us rely on.
On Friday this week the FTC’s open comment period regarding its “Policy Statement Addressing AI Accuracy” will close, which means that a bunch of very smart, very busy people are wasting a ton of time this week writing up comments that will mostly be ignored by the FTC — but still matter for the record. The whole thing is so ridiculous that a former FTC lawyer has sarcastically requested that the FTC publish a quarterly “schedule of values” so AI companies at least know which ideologies they’re required to support to keep Donald Trump happy.
It’s an unconstitutional sham from an FTC whose chairman, Andrew Ferguson, quite openly sees his job as putting his thumb on the scale of speech to favor the MAGA worldview. Ferguson has been doing this since the very start of his tenure and it doesn’t appear to be slowing down now.
He couches his policy and investigatory efforts in the language of legitimate FTC authority, but nobody’s really fooled about what’s going on. Here, when he talks about “objectivity and accuracy” in responses from AI engines, everyone knows what he’s actually doing is crafting a policy that will let the FTC punish AI systems for giving “woke” answers that the MAGA world disagrees with.
The mechanism at work is blatantly obvious: the FTC is taking its Section 5 authority over “unfair and deceptive” practices — generally meant to go after companies engaging in outright fraud or deception to trick consumers — to claim that if an AI’s output is deemed to be too woke or not pro-MAGA enough, then the FTC will accuse the company of being “unfair or deceptive” in its marketing.
The draft policy statement builds its whole case on the idea that users trust what AI tools tell them — which conveniently becomes the hook for worrying that those trusting users might get fed something MAGA world doesn’t like. Thus making it “deceptive.” Yes. Really. In the actual world, the FTC’s Section 5 deception authority requires that a company make a representation that’s actually false, and materially so. Here, the Commission simply asserts — with no evidence at all — what consumers “reasonably expect,” and then appoints itself the judge of whether any given output matches.
As they have marketed their remarkable breakthroughs to the public, AI companies have spent years representing explicitly and implicitly that their systems aim to produce the best output—output that faithfully and accurately achieves users’ stated objectives and the built-in objectives that users expect in the AI system—that is possible within their technological and resource constraints. Because of these representations and the inherent nature of the products and services in question, consumers have a reasonable expectation that AI systems aim to give truthful and accurate outputs. Consumers have no basis to believe that AI systems aim to produce outputs that are distorted by undisclosed ideological objectives.
Nonetheless, an AI company might be tempted to alter or steer the output of its systems contrary to consumers’ reasonable expectations for various reasons, including attempted compliance with a state law, such as Colorado’s recently revised Artificial Intelligence Act. But steering an AI system in this manner may deceive consumers in violation of Section 5 of the FTC Act. That is true even if the deceptive steering is done in an effort to comply with state laws. Of course, a company may be able to avert potential deception by making truthful, non-misleading representations about the aims of its model. But such representations would need to make clear that the AI company is prioritizing objectives different than those consumers requested or would otherwise expect.
This is all a bit of shadow puppetry, where the FTC wraps its “AI outputs should never be too woke” argument in language that pretends to fit a traditional FTC mandate.
But this is all wildly unconstitutional, as even a cursory reading of how the First Amendment works would show. As the Supreme Court recently highlighted in Moody v. NetChoice, internet companies have clear First Amendment protections in their editorial decision making regarding what they choose to show — or not show — users of their services. From that ruling:
…this Court has many times held, in many contexts, that it is no job for government to decide what counts as the right balance of private expression—to “un-bias” what it thinks biased, rather than to leave such judgments to speakers and their audiences. That principle works for social-media platforms as it does for others.
Yet, that’s exactly what this proposed FTC policy is setting up: if AI tools don’t produce properly MAGA-fied outputs, the FTC might go after them, claiming that the outputs are not in line with “consumers’ expectations” (as determined by the MAGA FTC) and thus, “unfair and deceptive.”
And while a “policy statement” from the FTC is not binding law, it’s clearly designed to publicly state what kinds of views will get you investigated by the FTC, in an attempt to create chilling effects that pressure AI companies to pre-censor their bots. This is also why the comment period is basically a formality. Ferguson has no obligation to do anything regarding the comments, as there’s no official rule being promulgated.
And don’t sleep on the FTC’s statement regarding Colorado’s (admittedly questionable) law, which seems to serve no real purpose other than to try to backdoor its way into Trump’s desire to magically block state AI laws, which is something he cannot unilaterally do. Remember, while there have been efforts in Congress to preempt state laws, that has not come to pass. But here the FTC is telling companies, in writing, that complying with an enacted state law creates federal liability exposure, entirely because the FTC policy (not even a full rulemaking) says so.
Last week we had former FTC lawyer Keith Fentonmiller lay out how obviously unconstitutional all of this is. It’s the FTC trying to dictate editorial policies of private companies. The First Amendment does not allow that. Aaron Rieke, another former FTC lawyer, put it even more starkly (and hilariously) in a recent LinkedIn post, designed to look like a letter in response to this open comment period, but which cuts through all the bullshit and says, in effect, “look, if you want us to only push the preferred ideology, can you at least tell us which talking points we should bless, and which we should suppress”:
Dear Commissioners:
I write in enthusiastic support of the proposed policy statement, and with one modest request for clarification.
The statement wisely prohibits steering AI outputs toward undisclosed “ideological objectives” while preserving companies’ freedom to implement “prudent guardrails.” As a consumer who relies daily on these systems — having been assured, deceptively it now seems, that they are “helpful” — I confess I cannot always tell these apart. The distinction appears to reside not in companies’ conduct but in the values they pursue.
I therefore respectfully request that the Commission publish, and update quarterly, a schedule of values, each designated either “Ideology” (deceptive if undisclosed) or “Common Sense” (no disclosure required). The proposed statement offers a promising start — “equity” is evidently Column A, while cybersecurity occupies Column B — but leaves substantial compliance uncertainty regarding, e.g., deference to law enforcement, patriotism, and politeness.
Absent a complete schedule, companies must simply guess which viewpoints the government currently disfavors and speak at their peril. I assume the Commission has already concluded that a federal schedule of approved and disapproved values raises no First Amendment concerns. Publishing the schedule would helpfully memorialize that conclusion.
Such a schedule would also generate efficiencies for future administrations, who would need only swap the column headers.
Thank you for your leadership in ensuring that American AI remains free from government influence over its viewpoints, as determined by the government.
Respectfully submitted,
A Consumer, Acting Reasonably in the Circumstances
While sarcastic, it makes the point better than any of the earnest comments will. An FTC that can punish AI tools for failing to parrot the administration’s ideological preferences is an FTC acting as a censor, and we’d all be a lot better off coming out and saying so, rather than pretending there’s some legitimate intent or purpose behind this effort.
Ferguson’s FTC has been focused almost exclusively on abusing the power of the Commission (remember, Donald Trump fired the Democratic Commissioners and has made zero effort to replace them despite the law requiring two commissioners from the minority party) to win culture war arguments and punish those deemed insufficiently loyal. The new policy and comment period is just more of the same. It’s entirely about Trump & Ferguson setting the sloppy groundwork for them to whine and complain about AI tools accurately calling bullshit on MAGA propaganda as being “unfair and deceptive.”
None of this should be happening. It’s an attack on the First Amendment so obvious that the FTC isn’t even bothering to disguise it well. But, because of the political world we live in today, everyone has to pretend to take it seriously, to pretend that the FTC will read their comments carefully, weigh the pros and cons of various approaches on this policy, and come out with some final policy that people should take seriously.
The FTC has no business investigating the editorial judgments of companies, and its facade about consumer expectations and deceptive practices is a joke. People and organizations ought to still submit comments, if only to establish opposition to this farce on the record. But what a waste of time and brainpower from people who have approximately a thousand more productive things to do.
Persuasion plays a key role in society. Whether it is political or financial decisions, workplace or family choices, or simply reading a book or article (like this one), often someone is trying to persuade someone else to agree with them, possibly by changing their mind. This raises an interesting question: if persuasion is such an important part of life, how good are the latest AI systems in this domain? Are they, for example, better than humans? That is what a research project has just investigated, and on an impressively large scale:
in a series of four preregistered experiments (n = 18,978 conversations from 6,923 people), we pitted AI systems against a range of human persuaders, including laypeople, winners of a separately preregistered four-round online persuasion tournament, professional canvassers, and world championship debaters.
The results were unequivocal:
We found that AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses. In a follow-up study, AI’s advantage persisted after experts received a coaching tool that let them practice against the AI that beat them, review their performance history, and see what AI would have said at key moments.
An arguably more demanding test found that AI systems were not just persuasive when it came to opinions, but also in terms of real-world actions: they managed to elicit substantially more real-money donations to charity than well-paid professional canvassers. The researchers were able to pin down the two key factors that helped AI to out-perform the best human persuaders in all these tests:
We found converging evidence that AI’s advantage stemmed from rapidly deploying larger quantities of information: after coaching, expert humans could tie an AI constrained to respond at human speeds and with human-length messages.
That is, AI systems were more persuasive largely thanks to the range of knowledge they could demonstrate, and the speed with which they could present it — precisely those aspects of AI that are improving all the time. Which means that frontier AI systems are likely to become even more persuasive in the future. That sounds a rather bleak prospect, but a commentary from Tom Stafford, professor of psychology at the University of Sheffield, and co-author of the book Mind Hacks, points out that things may not be as bad as they seem:
fact-based persuasion may indeed be effective, but that is good news for human reasonableness, not bad. The way the AI works isn’t some sinister magic; if it produces more facts, it is more persuasive. The constraint that persuasion requires evidence means that what anyone can be persuaded of will ultimately ground out on what can reasonably be claimed about reality. If AI is a tool which produces better-informed citizens and more respect for facts, that can be a positive thing.
That may be true in general, but the original researchers note that there are other factors at play here. For example, access to resources is clearly important:
power could flow to whoever can most readily access and deploy the most capable systems. In practice, that could mean the actors who already command the most resources, such as large private corporations, political campaigns, or nation states. These actors spend heavily to influence public opinion and consumer behaviour, and although the per-message effects of such efforts can be modest, such AI could raise their effectiveness, deepening existing imbalances in who can sway the public.
Another issue is that the persuasive power that comes with the deployment of leading AI systems could increase the clout of top AI companies:
in persuasion contests where both sides can secure access to the most capable systems, such AI could consolidate power by giving significant leverage to the actors that build and control those systems. These actors could tilt the outcome of such contests by, for example, deciding which positions their models will, and will not, argue for. In this case, power would flow not to the users of persuasive AI but to its suppliers, and consolidation of their influence would occur even when access among users is perfectly equal.
More positively, the researchers point out that as constant improvements in technology push down the cost of using persuasive AI
it could help under-resourced actors (e.g., pro se litigants and public defenders, small charities, grassroots activists) compete against more established and better-funded rivals, narrowing long-standing gaps in access to justice and assisting civic advocacy more broadly.
In his blog post, Stafford mentions another factor to consider:
In a world where every surface becomes filled with persuasive text, I don’t think it is inevitable that people will open themselves to being pulled in every direction. Not only do people have a significant degree of native scepticism, tending to resist persuasive efforts as they seek to maintain stability in their existing views, but they also have agency to open themselves, or not, to persuasive effects. The studies reported in this paper asked for an average of 14 minutes of conversation from participants. 14 minutes of sincere engagement might be a lot more than most of us give to alternative points of view in our daily lives.
In other words, we don’t really know yet what impact these highly-persuasive AI systems will have on politics, business, and everyday life. But given their superior ability to convince it seems likely that we will be encountering them more frequently in their role of indefatigable persuader, whether we want that or not.
