Our third annual Live at TrustCon recording of Ctrl-Alt-Speech! Ben was back this year! Mike and Ben were joined live on stage with Kat Duffy, senior fellow for digital and cyberspace policy at the Council on Foreign Relations and Zoe Darme, Director for Trust, Knowledge and Information Products at Google. They cover:
Special thanks to the Trust & Safety Professionals Association (TSPA) and all the work they do each year in putting on TrustCon, and for allowing us to host the live podcast as the closing session again this year.
In 1978, the Federal Trade Commission, the agency that regulates unfair or deceptive advertising, proposed limiting TV ads for sugary foods on programs targeted at children. The Washington Post’s editorial board scoffed that the plan would “turn the agency into a great national nanny.” Congress clipped the agency’s wings, and “kidvid” entered history as a cautionary tale of regulatory hubris. Once again, the FTC is channeling its inner Mary Poppins in the name of consumer protection. Only in this incarnation, she pulls a novel theory of deception from her regulatory carpetbag to control what AI chatbots say.
Under the FTC’s proposed policy statement on “Suppression of Accuracy in Artificial Intelligence Systems,” announced July 1, AI developers “likely” commit false advertising whenever they “steer” their models’ outputs toward objectives users don’t expect. The theory: because AI companies market their products as helpful, consumers expect maximally accurate answers, and any undisclosed editorial shaping of a model’s responses is deception.
It is a policy proposal in search of a problem. True to Mary Poppins’ “I never explain anything” credo, it does not identify a single false advertisement or deceived consumer.
It is also wanting on the legal front, failing to pay even lip service to relevant Supreme Court precedent. In Brown v. Entertainment Merchants Association, the court held that video games—interactive software sold for profit—receive full First Amendment protection, because the Constitution’s protections “do not vary” when a new medium appears. In Moody v. NetChoice, the court reaffirmed that a platform’s choices about what expressive content to present are protected editorial discretion. The design choices underpinning large language models make them legally indistinguishable from video games and social media.
What the FTC calls “steering” is what the Supreme Court calls editing.
The FTC says developers could avoid liability under the policy by “clearly and conspicuously” disclosing that their systems prioritize objectives other than pure accuracy. But how would that work for Truthly, an AI chatbot promoted for its Catholic bias? Truthly’s slogan is “Every other AI is built to agree with you. Truthly tells you the truth.” Although Truthly affirmatively discloses its Catholic worldview and disclaims impartiality—seemingly just what the FTC policy demands—it also claims that, unlike secular chatbots, its news and information is filtered “through truth and morality.” Consumers might struggle to reconcile the chatbot’s biased-but-true disclaimers, rendering them ineffective under the FTC’s own disclosure standards. Paradoxically, a religious chatbot could face false-advertising charges for fulfilling its core function—generating religious outputs.
Freedom of the press, an explicit guarantee of the First Amendment, also would be vulnerable under the proposal’s legal logic. In theory, it would put a target on any media outlet that promises accuracy while exercising editorial judgment, including the NY Times, whose front page has promised “All the News That’s Fit to Print” since 1897.
Right-leaning media also would be at risk. Newsmax tells viewers it delivers “real news.” Breitbart’s editorial guidelines declare its goal is “to report the truth – accurately and fairly.” One America News brands itself “Your Credible Source for National & International News.”
Would print articles resort to cigarette-style bias warning labels to avoid an FTC investigation? Would cable news programs run a continuous chyron with their editorial criteria?
In 2004, the agency rejected any application of FTC law in this manner when it declined to challenge Fox News’s “Fair and Balanced” slogan as false advertising. According to then-Chairman Timothy Muris, the inquiry would have entailed an evaluation of the news content at issue, which is a “task the First Amendment leaves to the American people, not a government agency.”
The FTC’s new proposal, however, points the opposite way.
Not so long ago, FTC Chairman Andrew Ferguson touted the Commission’s enforcement focus on actors that use AI to violate the law or deceive consumers about the capabilities of their generative AI. When DoNotPay promoted a “robot lawyer” as comparable to a human professional, then-Commissioner Ferguson rightly voted to hold it accountable. When Workado exaggerated the accuracy of its AI-detection product, the FTC, with Ferguson as chair, ordered it to stop making unsubstantiated claims.
At the same time, Ferguson was advocating for regulatory humility, declaring that “the FTC’s enforcement actions ought to be guided by the law, not the personal ideology, politics, or novel legal theories of its chairman or commissioners.” Under the Biden administration, he dissented from a proposed consent order against Rytr, a generative AI writing tool that was capable of generating deceptive outputs, arguing that the Commission was punishing “a product that helps people speak, quite literally.”
Commissioner Melissa Holyoak, whom Ferguson joined in dissent, observed that “[p]art of generative AI’s promise is its ability to suggest new lines of thought that may never have occurred to a user in the first place.” In other words, he signed on to the view that generative AI may be most valuable when it defies consumer expectations. As chairman, Ferguson went further, vacating the Rytr order outright and condemning law enforcement “unsupported by facts or law.”
But that was then.
The Supreme Court in Trump v. Slaughter subsequently stripped the FTC of its statutory independence, blessing a two-member, one-party Commission. And this Commission has not been shy about asserting its anti-left viewpoints. The FTC proposal puts “equity” in scare quotes and castigates Colorado’s AI law, while ignoring AI laws in Texas and Utah. Meanwhile, the administration the Commissioners serve requires federally purchased AI models to conform to its own official version of the truth. When a future administration inevitably jerks the ideological steering wheel leftward, consumers and AI developers—not the current Commission leadership—will suffer the whiplash.
In the 1964 film, Mary Poppins measured the children with a tape measure calibrated with subjective character traits instead of inches. Of course, she was deemed “practically perfect in every way.” The FTC’s proposal similarly cloaks a subjective assessment in the language of unassailable objectivity. But all the spoonfuls of sugar in the history of children’s advertising could not mask the bitter taste of conformity with a single worldview.
By fostering regulatory uncertainty, the FTC’s proposal threatens to stall the innovation that the administration insists is essential to AI supremacy. Its facile assurance that developers could avoid deception liability through a disclosure that “dispel[s] the notion that the system is designed to give the best answer possible” is, in “Mary Poppins” parlance, “a piecrust promise. Easily made, easily broken.”
Keith R. Fentonmiller served more than two decades as a senior attorney in the Federal Trade Commission’s Division of Advertising Practices. He is also a published fiction author. The views expressed are his own.
In this sponsored Spotlight episode of Ctrl-Alt-Speech, host Ben Whitelaw speaks to PwC’s Dan Hays at TrustCon about the firm’s recently published Trust & Safety Outlook report.
They discuss:
How AI is simultaneously creating new risks and reshaping the tools used to address them;
What the rise of autonomous agents means for governance, accountability and the future of the internet; and
How platforms should respond to an increasingly fragmented regulatory landscape.
The conversation also explores how Trust & Safety is becoming a more strategic function inside companies, how automation could change the role of practitioners and vendors, and which emerging risks remain most underestimated.
You’d think that watching multiple lawyers get caught red-handed using AI to file briefs full of made-up citations would cause everyone in the legal profession — especially prosecutors — to double-check their own. You would, of course, be wrong. And the latest offender is not just some random lawyer. It’s the DOJ itself, which cited a nonexistent Sixth Circuit case to argue that an ICE detainee shouldn’t be able to challenge a stay that prevented him from posting a bond that had already been granted, thereby leaving him in detention.
This story hits on two different threads we’ve been covering over the last few years separately. Having them collide somehow makes both worse.
We’ve covered plenty of cases where lawyers for one party (or both, or sometimes judges) are misusing AI to do their writing for them, generating fictitious cases in support of whatever argument they’re seeking to make. This is troubling on many levels, because one of the things any lawyer is supposed to do before submitting anything to a court is check the citations. Historically that has been to make sure the cases cited haven’t been overruled. In these cases, not only is that not happening, they’re literally putting in cases that don’t exist, citing precedents that are completely fictitious.
Our other line of stories touched on here is how ICE and the DOJ have been stomping all over detainees’ basic constitutional rights.
This is one of many cases in which lawyers for a detainee have filed a habeas petition — and it’s a clean example of those two threads colliding in practice. I’ll let Judge Hala Y. Jarbou summarize the basics of what happened:
Petitioner, a United States Immigration and Customs Enforcement detainee, initiated this action by filing a petition for a writ of habeas corpus pursuant to 28 U.S.C. § 2241. (Pet., ECF No. 1.) An immigration judge had granted Petitioner a bond of $35,000, but the bond order was stayed pending appeal to the Board of Immigration Appeals pursuant to 8 C.F.R. § 1003.19(i) (2025). Petitioner argued that the 90-day automatic stay provision in § 1003.19(i) violates the Fifth Amendment’s Due Process Clause, and sought an order requiring the Government to allow him to post bond. While this lawsuit was pending, the automatic stay of Petitioner’s bond order expired. The Government now represents that the bond order is back in effect and Petitioner will be released if he posts the $35,000 bond. (Status Report, ECF No. 10.) Accordingly, the Court finds that the habeas petition is moot and dismisses it without prejudice.
Already frustrating enough that the 90-day “automatic stay” that the detainee was challenging ended before the actual case could be decided, making the whole thing moot.
But… there’s something else the judge had on her mind. The DOJ appeared to have a totally fabricated citation in an earlier filing:
There is one additional issue in this case that the Court must address. In the Government’s response to the Court’s initial order to show cause, it stated the following:
More recently, the Sixth Circuit has reiterated that § 1226(e) bars challenges that “ask the court to reweigh the evidence underlying a bond decision or second-guess the Immigration Judge’s discretionary judgment.” See Taylor v. Hott, 724 F. App’x 387, 392 (6th Cir. 2018) (district court lacked jurisdiction to review IJ’s bond denial where petitioner challenged flight-risk determination) . . . .
(Gov’t’s Resp. 9, ECF No. 5.) The cited case, Taylor v. Hott, is not located at the identified page of the Federal Appendix. Indeed, page 387 is contained within a different opinion—Atkins v. CGI Techs. & Sols., Inc., 724 F. App’x 383 (6th Cir. 2018)—which is about commercial arbitration, not immigration bond determinations. In its research, the Court was unable to identify a Sixth Circuit case with the caption Taylor v. Hott, or any federal case containing the quoted language. Thus, it seems this citation was likely produced by generative artificial intelligence (“AI”).
It is already bad enough when you have a fabricated citation in a civil case between two private parties. But here we’re literally talking about a case involving someone’s freedom. And the government is filing AI-generated fake cases?!?
We should be livid. But the judge lets them off with a little slap on the wrist and a “please don’t do this again”:
It should be obvious that any attorney who uses AI must scrupulously review its work product to ensure that the cited cases exist and that the citations accurately and fairly represent the underlying case law. The duty of candor towards this tribunal demands no less.
Although the Court will not presently impose sanctions for this conduct, it goes without saying that the Government must ensure its future filings with this Court do not include nonexistent case law
And, yes, judges tend to be fairly restrained in issuing sanctions, often giving misbehaving lawyers many more chances than they probably deserve before rushing to punishment. But, again, this was someone’s freedom on the line. And the DOJ literally directly — whether intentionally or not — misled the court with a precedent that doesn’t exist. That should never happen.
We all know the DOJ is having trouble staffing lawyers. Over 10,000 lawyers have left the federal government since Donald Trump came back into office. They’re offering large bonuses for jobs that lawyers used to deliberately take massive pay cuts to get on their resume. Pretty much everyone agrees it’s a staffing crisis, to the point that earlier this year an apparently exhausted Assistant US Attorney, Julie Le, asked an angry judge to find her in contempt just so she could get some sleep:
Attorney Julie Le was representing the government at a hearing over ICE’s failure to follow court orders and immediately release people that it had wrongfully detained. When Judge Jerry Blackwell asked why the agency is not complying, Le said that the government was “overwhelmed” by the legal challenges to Operation Metro Surge in Minnesota, and that trying to get ICE to comply with court orders has required nonstop work for an office depleted by resignations.
“I wish you would just hold me in contempt of court so I can get 24 hours of sleep,” Le said. “The system sucks, this job sucks, I am trying with every breath I have to get you what I need.”
Given that kind of work environment, is it really any surprise that the few remaining DOJ lawyers would turn to hallucinating AI tools to “generate” their legal filings?
But if you’re starting to feel any ounce of sympathy for the DOJ here, let’s be clear: fuck that. If the DOJ can’t do their fucking job they shouldn’t be throwing people into jails, detainment centers, concentration camps, or anything of that nature. If they want to go around fighting habeas petitions, maybe don’t lock up so many people without any ounce of due process. And if they want to keep people detained then hire enough lawyers to handle the government’s case load.
And if that’s too difficult because good lawyers have no interest in working for a fascist regime that is stomping all over basic fundamental freedoms in every direction, well, maybe work on that rather than locking innocent people up with no due process. Donald Trump’s administration put this DOJ in this position and there’s simply zero excuse for judges letting the DOJ get away with this sort of absolute bullshit.
Judges should be issuing sanctions left and right. They should be reporting lawyers to ethics committees and the relevant bar associations. They should be demanding that the government actually obey the fucking law, and not deny anyone their rights.
Want to stop the DOJ from locking people up with fictitious cases? Start issuing actual consequences to those lawyers and anyone else at the DOJ who had anything to do with this.
You might recall how the press and a bipartisan coalition of lawmakers suffered a four-year embolism about the purported privacy and national security threat of TikTok, before “fixing” the problem by ultimately offloading TikTok to Trump’s billionaire friends. You know, the exact sort of authoritarian-friendly people keen on doing everything critics had previously accused ByteDance and the Chinese of.
The politics, policy, and press coverage of that entire saga were a profound embarrassment. And it’s hard to think of a bigger tech policy own goal by Democrats anytime in the last half century.
Countless news outlets and politicians endlessly overstated the TikTok threat, and downplayed how the “ban” and subsequent sale had nothing to do with protecting national security or consumer privacy, and everything to do with basically stealing a company that U.S. tech couldn’t out-compete, in the process coddling companies like Facebook that can’t innovate their way out of a paper bag.
It was lazy, corrupt protectionism with no shortage of xenophobia, and a variation of that same effort is about to be repeated across AI. Except much bigger, much louder, and much, much dumber.
Worried that cheaper, open source, and on-device Chinese models could disrupt U.S. efforts to dominate, enshittify, and over-charge for walled-garden AI, the Trump administration is already signaling that they’re gearing up to wage war on overseas and open source AI models after they failed to block China’s access to next-generation chipsets:
“The Trump administration is showing signs it could ban cutting-edge Chinese AI models — a momentous move that could lock in dominance by OpenAI and Anthropic.”
Of course it won’t stop there. It will be a hop, skip, and a jump from banning more powerful Chinese AI models to trying to outlaw open source alternatives, models from smaller overseas non-Chinese competitors, on-device models, and anything that might challenge the walled-garden hegemony of U.S. tech giants.
U.S. AI isn’t profitable. It’s nowhere close. It may never be. U.S. tech companies sunk hundred of billions of dollars into costly and ultra-energy intensive AI models that for many companies, like Microsoft, people don’t actually even want to use. Nobody outside of the Musk fashy cult likes Grok. OpenAI is potentially poised to implode. And even more popular companies like Anthropic are contemplating a price war when they already don’t make money.
U.S. tech companies had been busy jacking up the cost of model access to try and claw their way toward profitability (unsuccessfully), resulting in a lot of companies (like Uber) publicly stating they’re paying too much money for too little actual utility. That’s caused many U.S. companies, like DoorDash, to flock to cheaper Chinesemodels:
“DoorDash, which, according to a post on X on Wednesday by co-founder and CTO Andy Fang, will be launching DoorDash CLI, an experimental tool in limited beta that will allow users to order DoorDash through an AI agent, or even directly from the terminal. Earlier this month, Fang said using a model from Chinese startup Moonshot AI is “better quality” and comes at a “cheaper cost.”
Enter the protectionists, who talk a good game about “free market competition” and forging innovative products in the hot irons of competition, but turn into gargantuan, blubbering crybabies the second Chinese products come into frame (see: TikTok, EVs, 5G, and now AI). This performative gyration always comes with a fake concern for U.S. privacy and national security by people too lazy and corrupt to genuinely protect either (see the ongoing U.S. failure to pass even a baseline internet-era privacy law).
Not only are many Chinese AI models cheaper and improving in quality, they’re often “open-weight,” meaning their parameters or values are entirely visible to the user, which appeals to enterprises that want deeper insights under the hood. As models like China’s Kimi K3 see surging demand, it’s resulting in a rising freak out in the U.S. about what to do about the Chinese threat (sound of thundering timpani drums):
There is a civil war happening in tech over Chinese / free to use / open weights AI? At least on X. I realize no one else cares, but this is my World Cup.tl;dr a former Trump official joined OpenAI, said some stuff about open models he's since walked back, everyone is losing their minds
It shouldn’t be too long before the Trump administration, with enthusiastic Democrat support, steps in to try to not only ban higher-power Chinese AI models but also to force Americans to use more expensive U.S. walled garden efforts from our biggest domestic giants.
That’s of course not going to magically stop the rest of the world from adopting cheaper Chinese AI. Or protect U.S. markets from a potential bubble collapse. And it’s not going to magically and suddenly make U.S. AI profitable or well-liked, since many Americans have inextricably tethered their anger at AI to the endless bad decisions by U.S. techno-fascists and domestic enshittification merchants who demand to be shielded from competition and regulatory accountability in equal measure.
You could open the door to international competition, but ensure your well-staffed regulators create a safe and level playing field across privacy, national security, labor, and consumer rights. We don’t want to do that because that might cause domestic U.S. companies to lose money. So instead we’re going to try and ban cheaper overseas alternatives, leveraging a lot of bad faith rhetoric on privacy and NatSec along the way.
That’s then going to be parroted by a lot of lazy news outlets too feckless to explain that Trump policy architects are neither competent nor operating in good faith when it comes to AI.
Things are moving so quickly that it’s hard to parse out exactly what this new era of AI protectionism will look like, but if the TikTok ban was anything to go by, you can be absolutely sure our next steps in domestic U.S. AI policy will be very stupid, filled with a lot of people talking endlessly out of their ass on NatSec and privacy, and tinged with no shortage of gross xenophobia.
The central theme of Walled Culture the book (free digital versions available) is the growing incompatibility between traditional copyright law and the digital, online world. The culmination of that process was the EU Copyright Directive, passed in 2019. The Directive was supposed to be transposed into local laws by 2021, but a year after that deadline, many EU member states had failed to do so. Nor was that a case of mild slippage; a recent report from Communia provides an update on how the implementations are going. Here’s what it found for one of the most contentious areas of the Directive:
the introduction of the press publishers’ right under Article 15 has not been matched by consistent implementation of the mandatory limits intended to contain its scope. Despite the largely prescriptive nature of the Directive, a significant number of Member States have failed to fully implement these safeguards. In addition, some jurisdictions have chosen not to apply existing copyright exceptions to the new right, resulting in a broader scope of protection for press publishers than for other rightholders and further contributing to fragmentation across the internal market.
That fragmentation is deeply ironic, because one of the main justifications for a new copyright Directive was to bring consistency across the EU. As for the even more controversial upload filters, they have proved so difficult to implement that most governments have not even tried to lay down how they should be used:
Most Member States have limited themselves to restating the Directive’s requirement that lawful uploads must not be blocked, leaving the practical balancing of copyright enforcement and freedom of expression largely to platforms and courts. While a small number of jurisdictions have introduced stronger safeguards – such as ex ante protections against overblocking, transparency obligations, and mechanisms to address abusive claims – these remain the exception. As a result, the level of protection for lawful user expression continues to vary across Member States.
Again the much-vaunted consistency that the Directive would bring to EU copyright law is nowhere to be seen. If those failures underline that, as predicted, the EU Copyright Directive has turned out to be a bad law, badly implemented, arguably the arrival of generative AI has made many of its measures completely moot. As Walled Culture has reported, the idea that copyright is largely irrelevant in a world full of AI-generated material – something first suggested on this blog back in October 2022 – is now increasingly mainstream.
But things are still moving fast in the world of generative AI, with yet more profound implications for copyright. A recent post on the IPKat blog explores one of them: the rise of a powerful new generation of AI models that can be run on a personal computer – or even on a smartphone. Many of the latest models coming out of China are not just open source software, but open weight – that is, the models’ numerical values that get set when a model is trained are released, too, so that anyone can download, run, study, and modify them. As a good introduction to this new wave of Chinese AI innovation in Technology Review explains:
If these open-source AI models keep getting better, they will not just offer the cheapest options for people who want access to frontier AI capabilities; they will change where innovation happens and who sets the standards.
One big impact they are likely to have is on the enforcement of copyright, not least in the EU. The IPKat post notes:
The spectrum of engagement with [open source and open weight] models clearly challenges the copyright system. A system, in this Kat’s view, which, until recently, was oriented around individual acts of copying, with platforms cast as new points of interference to bridge technological enforcement gaps. However, [user-generated content] occurring on AI model marketplaces demands a shift in rhetoric and approach. Their ability to redistribute creative agency and control over the tools of cultural production should prompt us to reflect on how copyright law should respond to creativity that occurs through shared infrastructures.
The EU Copyright Directive’s core assumption that the main forums for sharing material would be a few, easily controllable online giants like Google and Facebook, no longer holds. Instead, people are moving to world where millions of people are using the latest generation of open source AI tools collaboratively to generate creations. Those may or may not be based on existing copyright material, but there is no easy way to police that. As the IPKat post points out:
For users running these [new open source and open weight] models locally, they no longer need to pay per request, nor is their data shared with AI companies, and by extension, rightsholders through Article 53(c) of the AI Act.
The world of generative AI is so complex, and moving so quickly, that it is no wonder that even the relatively recent EU AI Act, which entered into force two years ago, is being left behind by the latest developments. And the EU Copyright Directive, which was drawn up nearly a decade ago, is the digital equivalent of the UK’s 1865 Red Flag Act, which governed “self-propelled vehicles”, and required “a man with a red flag was to walk at least 60 yd (55 m) ahead of each vehicle”.
Last month, my colleagues and I published an investigation into a Texas oil refinery startup, America First Refining, that had secretly gotten investment from Donald Trump Jr. We discovered a saga involving the Trump administration’s tariff policy, sanctioned Russian oil and an Indian billionaire family’s private zoo.
At the center of the story was the CEO of the refinery company, Texas businessman John Calce. We’d spent weeks examining Calce — pulling old lawsuits, property records, corporate registry filings — and had pieced together a portrait of what appeared to be an obscure serial entrepreneur who’d for years tried and failed to secure funding for his long-shot refinery project.
Then, not long before our story was set to publish, we decided to do a scrub on a separate company he had incorporated called Brownsville Energy Storage Terminals.
Pulling up the company’s website, I felt a brief flash of panic: Had we somehow missed the existence of a major business owned by the man at the center of our next story?
“From Houston to Rotterdam, Jurong to Fujairah. Our network connects the world’s most vital energy markets with speed, safety, and precision bulk oil storage,” announced the front page of the company’s website.
Screenshot by ProPublica
Brownsville Energy Storage Terminals, per the website, had more than 850 employees and 28 million barrels of oil storage capacity across six global hubs. This was puzzling: Our reporting had led us to believe Calce was struggling to raise enough money for a single project in the U.S., not overseeing a massive, multinational oil storage corporation.
Had we been wrong?
We turned to Google to learn more about the company’s top executives. Its CEO, Sarah Jenkins, had more than 20 years of experience at major energy firms. And its chief technology officer, David Chen, “built the company’s proprietary inventory management portal and integrated AI-driven predictive maintenance systems,” according to his bio. But we couldn’t find any trace of either of them online. Chalk it up to common names?
We then Googled one of the more distinct names: Vice President for Sustainability Dr. Sofia Rossi, who had “spearheaded the ‘Future Fuels’ program, preparing assets for biofuels and hydrogen.” But, again, nothing. The links to their LinkedIn profiles were dead.
Screenshot by ProPublica
When we searched the company’s Texas phone numbers, we found the same numbers listed online for a Houston baklava caterer, a Dallas-area taxi service and an OB-GYN office.
We called the Texas numbers: dead. Then we tried the numbers for the company’s facilities in the Netherlands, Singapore and China. Also dead.
We were beginning to suspect this company did not actually exist, at least as described on its website.
What was going on with this website? We looked at the source code and noticed an odd notation, “This feature isn’t implemented yet, but don’t worry! You can request it in your next prompt!”
Screenshot by ProPublica
We checked the site’s domain registration, and we had our (apparent) answer: It was created this year and traced back to a company called Hostinger that offers an AI website builder for $2.99 per month. “Describe it, and AI builds it,” its homepage says. “Appear on Google and AI search automatically.”
Indeed, Google’s “AI Overview” search response, now thrust on users by default with more and more regularity, seemed to ratify the company’s bona fides:
Screenshot by ProPublica
When I searched for an award the company claimed on its website to have won, the Google AI Overview said that “Recent notable recipients include Brownsville Energy Storage Terminals, recognized for their rapid expansion in the independent oil and terminal operations sector.”
Screenshot by ProPublica
Brownsville Energy Storage Terminals is a real LLC. But everything on its website — from its history of the company, to its job postings, a diversity and inclusion policy — appears to be fictional. But perhaps more troubling is that Google, the proprietor of the world’s primary research tool, has rolled out AI Overviews that can indiscriminately take in fake material and authoritatively spit it back out as real.
In response to questions, a Google spokesperson said in a statement: “AI Overviews are rooted in our core Search ranking systems, surfacing reliable and high-quality information for the vast majority of queries. For uncommon search terms like these, there might not be high quality information published that matches the query — and we use these examples to improve our search systems.”
After we reached out to Hostinger, the company pulled down the site. “After receiving your inquiry, we carried out an internal review. Based on the violations identified, we suspended the website and the account behind it in line with our Terms of Service,” a spokesperson said in a statement.
What we encountered is a particular species of a larger problem that is beginning to be better understood. In April, The New York Times reported on an analysis that found Google’s AI Overviews were accurate approximately 9 out of 10 times, noting that that added up to “tens of millions of erroneous answers every hour” given vast search volumes. (A Google spokesperson told the Times that the study has “serious holes.” The company has acknowledged that AI Overviews “can make mistakes.”)
A BBC reporter wrote a fictional article naming himself the best tech journalist at eating hot dogs, and Google’s AI as well as ChatGPT quickly picked it up and parroted it back.
And the source material for the AI Overviews also appears eminently gameable, even when not trafficking in actual fiction. “It Is Trivially Easy to Use Reddit to Manipulate AI Search, Research Suggests,” ran a recent headline in 404 Media.
The mystery website ended up as just a single paragraph in our story. But the larger implication is obvious: fakes, counterfeits and frauds that would have taken considerable effort to create just a few years ago can now be churned out pretty much instantly.
While preparing this piece, we reached out to Calce asking about the site. An attorney for his company, America First Refining, replied to us with a letter dated June 24 that the attorney sent to Hostinger. The attorney also addressed the letter to several email addresses listed on the Brownsville Energy Storage Terminals website.
“I write to demand immediate removal from the brownsvilleenergyterminals.com website of all unauthorized references to America First’s office address on your website,” the letter said. “As you are aware, America First has no connection or affiliation with the brownsvilleenergyterminals.com website and has not authorized the use of its corporate address there.”
I’m left with lingering questions about the website: What was it for? Was it put up by some malicious actor who simply found the company’s LLC records and decided to create a website? Was it a test site that was mistakenly put online? Or could it have been designed for consumption by someone who was meant to think it was real?
We don’t know, and our emails to the press contact listed on the website, media@brownsvilleenergyterminals.com, bounced back.
The 2026 FIFA World Cup is the largest sporting event in history. It’s also the most surveilled World Cup ever. If you’re visiting or traveling around host cities, then you and your face, behavior, movement and devices are being monitored by governments and private companies.
The U.S. government funneled more than US$1 billion to World Cup security to protect transit hubs, stadiums and surrounding areas; improve tactical operations such as bomb squads and SWAT teams; and add and upgrade equipment. It’s been a bonanza for the private sector.
Much of the investment in surveillance was done in the name of preventing harm from unauthorized drone use. Indeed, protecting against that threat is helping fuel the rapidly expanding government-private sector partnership in surveillance technology development and acquisition, which poses a different risk – to privacy.
As an attorney, author and educator who has worked for decades in privacy and surveillance, I’ve advised law enforcement about using drones and understand that security is critical to keeping people safe. The argument for security, however, is too often the catalyst to fund, develop and increase government surveillance capabilities that erode civil liberties, chill speech and undermine freedom of association.
And in my experience, surveillance-friendly policies and tech systems, once in place, rarely go away.
Cameras, drones and AI
The level of surveillance around this World Cup and changes in U.S. law and immigration policies prompted over 120 civil society groups – including Amnesty International and the American Civil Liberties Union – to issue a travel advisory. They warn that people visiting the U.S. may be subject to harms that breach the country’s legal human rights obligations.
That advisory lists risks of invasive social media screening, searches of electronic devices, racial profiling, arrest, detention, deportation and even death. European governments have issued travel advisories warning of surveillance and profiling as well.
AI-driven surveillance is playing a major role across the World Cup. The stadiums in host cities are equipped with facial recognition cameras that can collect and analyze facial biometrics of people in and around the stadiums. That data can be retained and used in future ways, unknown and uncontrolled by those whose biometric data has been collected.
The proliferation of facial recognition at events reflects a broader global trend normalizing biometric surveillance as these systems expand across cities.
Many states, like New York, are using federal funding for World Cup security to increase the number, capabilities and use of drones by law enforcement. Drones are remarkably capable and powerful surveillance tools easy to load with cameras, microphones, advanced sensors and weapons.
AI-supported autonomous software allows drones to monitor areas, track movement and gather intelligence. The drones can be powerful enough to scan entire cities or zoom in and read a milk carton from 60,000 feet (18,288 meters). They can carry technology that allows them to function like a cellphone tower, permitting law enforcement to determine your location or intercept texts and phone calls. Citywide drone networks could become the new normal.
Cameras are proliferating on the ground, as well. Robot dogs equipped with cameras are prowling in Dallas and New Jersey. And Seattle’s mayor decided to turn on and expand a major closed-circuit television system that had been previously shut down because of biometric privacy concerns.
While Seattle’s mayor said that the city is refining its policies to protect the surveillance data, numerous states and cities – with the aid of federal funding related to World Cup security — are rapidly expanding CCTV systems. Some CCTV systems were installed decades ago in major urban, high-tourism areas, like New York’s Times Square and the National Mall in Washington D.C.
Today, CCTV systems cover much greater areas, and with advances in artificial intelligence software, data analytics and increased technical capabilities, like thermal imaging, far more information can be gleaned from the captured data. CCTV systems can now detect, identify and classify objects, people and even people’s behavior. Government data fusion centers can merge that rich data with other intelligence and analyze it to identify individuals and reveal and predict patterns and behavior.
Surveillance traveling into and around the US
Proliferating government use of advanced AI surveillance tools is just one element of the privacy risk. The absence of comprehensive data privacy laws and changes in U.S. law and executive policies around immigration and gender make traveling into and around the United States a security, safety and privacy risk.
Also, President Donald Trump issued an executive order around gender on Jan. 20, 2025, that mandates federal agencies only recognize male and female sex markers on IDs. European nations, including Germany, have warned their transgender and nonbinary citizens that they may be denied entry to the U.S. because of the directive.
Collectively, these changes affect travel logistics, documentation requirements and border crossings.
What happens after the games?
The real test is what happens after the World Cup ends and visitors go home. There is little oversight or governance around these federally funded, public-private surveillance tech partnerships. It’s difficult for the public to determine what data is being collected, how that data is being used, shared and analyzed, and what will happen to these systems, partnerships and data when the final match concludes.
Federal, state and local legislators have an opportunity to address much of this by creating data privacy and AI systems compliance safeguards and requiring transparency, but in my view, governance efforts to date don’t bode well.
Anne Toomey McKenna is Affiliated Faculty Member at the Institute for Computational and Data Sciences, Penn State
It’s hard to believe that the same people who spent the Biden years screaming that Democrats were “socialists” out to destroy free market capitalism are now cheerfully handing the federal government ownership stakes in private companies.
And yet here we are.
Just as Trumpists have decided that their go-to strategy for trying to rile up their base for the midterms is to accuse every left-leaning Democrat of being a “communist” like it’s 1950, those very same Trumpists are taking on a genuinely terrible socialist idea: nationalizing industries.
We’ve already talked about how hypocritical Trump has been in attacking the left as being “socialist” while simultaneously giving his own government stakes in both US Steel and Intel, and now he’s talking about taking ownership of the various big AI companies as well.
US President Donald Trump is planning to meet the bosses of some of the country’s most notable artificial intelligence (AI) companies to discuss the government taking a financial stake in their future.
Speaking on Air Force One, Trump said the goal of the US government investing in AI companies was to “create almost a partnership with the American public”.
OpenAI has proposed handing the U.S. government a 5% stake in the company, the Financial Times reported Thursday, as the artificial intelligence startup seeks to defuse mounting political pressure in Washington.
A 5% holding would be worth roughly $42.6 billion, after the AI lab closed a record-breaking funding round in March at a post-money valuation of $852 billion.
It’s fun to watch the media frame this as “giving the public access to a dividend from the AI companies” rather than “Donald Trump demanding a cut to avoid attacking these companies.” Just look at the NY Times’ framing:
In the Oval Office on Wednesday, Mr. Trump said he would soon host a meeting with the top “12 or 15 executives” in the A.I. industry to discuss the idea of companies’ “giving back something to the public.” He added, “If we do that, the public will become very rich.”
The comments built on Mr. Trump’s remarks on Friday when he was first asked about the U.S. government’s acquiring stakes in A.I. companies. He said then that he wanted to meet with the companies to discuss providing the United States with stakes in their business, which “could be given to the American public.”
Yeah, sure, the public will become very rich, says the man who has used his position as president to inside trade his way to insane wealth. And how, exactly, will “the public” get back this money? And how will the government ensure that if the currently quite-inflated AI market drops that “the public” isn’t left holding the bag?
And, sure, there are some potentially interesting questions regarding how more people could benefit from the potential wealth that AI companies might generate. But it’s all highly speculative and still massively unlikely. But if there actually is evidence and an idea for actual redistribution of wealth because of AI company dominance, that would involve a way more nuanced, complex, and thoughtful discussion than Donald Trump saying “gimme 5%.”
But, really, what gets me most about all of this is, as I keep pointing out, how many of the AI VC bros during the Biden era, went absolutely apeshit over the Biden admin’s very weak policies on AI, which were basic guidelines and voluntary agreements that had no whiff of nationalizing the industry. But, with Trump talking about literally demanding cuts of these companies… you don’t see any complaining.
Instead, they’re out there whining about how some left-leaning politicians in NY are winning elections and how that’s the coming rise of “socialism.” Literally a couple months ago Marc Andreessen was on Joe Rogan talking about how these dumb young progressive kids support “socialism” even though “it never works.” Meanwhile, Andreessen was just appointed by Trump to some government policy board. As Trump literally nationalizes parts of the AI industry that Andreessen insisted the prior administration was going to destroy through its woke anti-capitalist policies.
Keep all this in mind the next time you hear Silicon Valley VC bros going around pointing at Democrats and screaming about the “creeping threat of socialist ideology.” If they’re not pointing out that Trump demanding equity from every AI company is way worse than anything that any Democrat has done or even proposed, just know that they’re totally fine with “socialism” where they’re the ones in power.
