From the very beginning of this recent obsession with identifying everyone online (yes, they like to call it “age” verification, but it always ends up as identity verification), we’ve been pointing out that it was a huge privacy nightmare waiting to happen. Or maybe it wasn’t waiting. Maybe it was already happening.
This week a massive new data breach has been revealed that should put the nail in the coffin for the idea that any sort of age or identity verification could be safe. 153 million scans of drivers licenses easily available based on this breach, with more being added all the time. Literally on the day it was revealed (and right before the site was taken down) it added another 400,000 records to its available database.
There is no safe age verification. There is no age verification that doesn’t put people at risk.
Last year, Eric Goldman wrote the definitive piece on how all of these technologies — no matter what they tell you — are huge privacy risks, but people are still living in denial. This is despite the numerous examples we’ve had in just the past few years of verification providers and their customers having massive data breaches.
The latest comes to us via Brian Krebs, who reports on a massive breach of scanned IDs — more than 153 million drivers licenses from people across the US and Canada, now for sale on the dark web:
A new identity theft service launched on the dark web this week is selling digital scans of more than 153 million drivers licenses from people in the United States and Canada. Based on interviews with individuals whose licenses are available for purchase on this service, it appears to be siphoning images collected by a widely-used identity verification company based in Louisiana. KrebsOnSecurity also has learned that the New Orleans field office of theFederal Bureau of Investigation(FBI) today launched an official inquiry into the source of the images.
Krebs traces the breach back to an ID verifier that appears to be used by many companies, including Hertz, the rental car company. It appears to not be limited to them either, as he checked with a number of people who were in the database, and by looking at the date they were added alongside their calendars, found examples of other people who shared their ID at places like a pot dispensary.
That company turns out to be IDScan.net, based in Louisiana, which has contracts with thousands of dispensaries, not to mention Hertz, FedEx, and Target. And while Krebs is focused on how many of the leaked IDs are connected to real world businesses, it’s worth noting that IDScan.net is also doing age verification for a bunch of tech companies, has a page tracking state age verification laws and company implementations, and even has written positively about laws like KOSA, the Kids Online Safety Act, that would effectively require age verification.
So, yes, we have a company that is a big player in the age verification space, talking up age and identity verification laws, that appears to have had a long-standing ongoing leak of every ID it scanned.
Yiiiiiikes.
And, of course, like all age and identity verification providers, IDScan has spent years talking up how secure it keeps all this data, even as every single record appeared to be leaking in realtime. Here’s their “Trust Center” page which is still up days after the hack was revealed:
That’s the company that spent over a year leaking 150 million drivers licenses in real time, explaining “how we protect data, maintain system reliability, and earn the confidence of our customers and their users.” Might be time to update that page.
But also, this should be a massive warning to everyone pushing for age verification laws. You can have a “trusted” company in the space who brags about all the certifications it has. It’s in “compliance” with the GDPR, the CCPA, and every other law. It is “transparent” about its “privacy practices” and how its “sensitive identity data is handled responsibly” and…. for over a year it’s been leaking all of those sensitive records.
And it appears no one internally at the company noticed.
As Krebs makes clear, the breach included many, many millions of records and ID scans that were being swiped in real time by the hackers who breached the system:
The people behind Nexus claim the license images are coming from an active breach at “a major identity verification company” whose customers include multiple Fortune 500 companies.
The record totals listed by the Nexus identity theft service. The number of drivers license records increased by nearly 400,000 in the span of just 24 hours.
“We have been continuously exfiltrating new data for over a year into our private database,” the service enthused in its introductory post on Exploit. “Records are available to preview before purchase with pertinent information redacted. Customer photos are displayed if available.”
Indeed, over the past 24 hours, the number of drivers license records listed as available in Nexus has increased by nearly 400,000, suggesting that freshly stolen license data is being harvested and uploaded to this serviceon a semi-regular basis.
And the exposed records aren’t just random members of the public. Krebs found the driver’s license of the sitting Secretary of Defense sitting in there for sale:
A bargain! Only $100 to get a scan of the Secretary of Defense’s driver’s license.
Anyway, each time we highlight a breach people play it down and insist that mandating age verification is perfectly safe and nothing to worry about. Yet here’s one of the largest identity verification companies in the country, with a pipeline so wide open that hackers had a real-time feed of every government ID it scanned, for over a year, without anyone at the company noticing.
Krebs spoke to a security researcher at Cybera, named Larry Baldwin, who talks about how this kind of data can do real damage:
Baldwin said the Nexus identity theft service presents multiple serious security and privacy threats, noting that state-issued drivers licenses are commonly used as proof of one’s identity when opening new lines of credit. Baldwin said the service could also dangerously expose many people who do not wish to be found but who cannot meaningfully change their appearance (or at least not enough to fool today’s AI-based image matching tools).
This category of people, he said, includes those fleeing domestic violence, and even people who have been assigned a whole new life and identity as part of the federal government’s witness protection program, which is generally reserved for criminal defendants in racketeering and conspiracy investigations who agree to cooperate with federal authorities.
“Just when it seems like we’re making some headway in improving authentication controls through drivers license verification systems, this happens and the very thing those improvements are dependent on are compromised,” Baldwin said.
At this point, anyone still supporting age verification requirements, especially claiming it’s for “child safety,” should have to answer for all the millions of people put needlessly at risk due to data breaches like this.
You cannot do age or identity verification safely. It always creates some sort of record and that set of records will always become a target. That’s what happened here. And it’s what will happen with any such systems.
Meta’s settlement with 52 state attorneys general is a bad deal for all internet users, and especially for teens. That’s what we said the day the settlement was announced.
In this post, we go through the Settlement’s provisions in detail and explain why that is so, including how:
The Settlement embeds age assurance technology and age-gates into Meta’s social media products and requires all users—minors and adults—to undergo a rights-threatening age estimation process
The Settlement places severe restrictions on Teens that can only be modified by the Teens’ parents and only then in exchange for giving their parents a ton of information about their online community and usage;
The Settlement seems to empower the attorneys general to enforce Meta’s content restriction on “age inappropriate content,” categories that Meta itself has had trouble administering without excluding information about sexuality, sexual and reproductive healthcare, and abortion medication;
The Settlement actually requires Meta to collect, analyze, and retain more information about its Teen users—when the pressure should have been on Meta to diminish its surveillance capitalism.
Note: A chunk of the settlement addresses unauthorized users under age 13, and Meta’s obligations to comply with the Children’s Online Privacy Protection Act. Meta policy has banned users under 13 since the company opened to the public in 2006. Aside from the age assurance frameworks that support both those and the other parts of the Settlement, the under-13 provisions are not addressed in this post. Those provisions essentially require Meta to detect and delete all under-13 accounts.
Further note: All U.S. states are parties to the Settlement except Florida, New Mexico, and Texas. The Settlement includes D.C., American Samoa, Guam, Northern Mariana Islands, and Puerto Rico.
Age Gates Reinforced By Age Estimation Technology
In the Settlement, Meta agrees to age-gate Instagram and Facebook, thus making age gates a legal mandate. And Further, Meta will now enforce these age gates with “age assurance” technology, ditching its previous practice where the person signing up for the services self-attests to their birthdate. This concession firmly embeds deeply flawed age estimation technology into the online experience of millions of people around the world. First and foremost, the age verification setup seriously threatens online anonymity and privacy for everyone, as we’ve said before. The Technology also just adds a layer of creepiness into the user of any service. In the Settlement, Meta pledges to, within one year, apply one or more age assurance methods to each Instagram or Facebook user in the states and territories that joined the Settlement. [P. 10, §II.A.1]
1. Age Assurance Framework. Within one (1) year of the Effective Date, Meta will adopt an age assurance framework (“Age Assurance Framework”), wherein it will apply one or more age assurance methods developed by a third party and licensed to customers (“Commercially Available Age Assurance Methods”) orage assurance methods developed by Meta (“Proprietary Age Assurance Methods”) (collectively, “Age Assurance Methods”) to each Meta SMP user in the Settling States. For the purposes of this Section II.A, an age assurance method developed or acquired by Meta that uses the same or functionally identical technology and methodology to a Commercially Available Age Assurance Method shall be treated as a Commercially Available Age Assurance Method. The Age Assurance Framework must include Age Assurance Methods to evaluate whether a Meta SMP user is a Teen User or U13, as described in Section II.A.6. New users of Meta SMPs who have not yet had their age assessed by an Age Assurance Method pursuant to Meta’s Age Assurance Framework shall receive the Default Protections pursuant to Section II.A.10 of this Agreement.
Those methods might include both commercially available products, as well as proprietary age estimation process Meta might have or develop. Meta also pledges to consider age signals from Google and Apple operating systems and app stores. [§II.A.5] Meta has previously advocated for age assurance requirements to fall on Google and Apple rather than on individual services.
This age assessment essentially dumps users into one of three age-range buckets: 18+, 13-17, and under-13. Users under 13 have long been barred from Meta products, but this Settlement creates new obligations to search for and detect users who may have said they were older.
For those estimated to be over-18, the Settlement guarantees no direct benefit to you: no privacy protections, no greater user controls for your own accounts, no dent in Meta’s surveillance capitalism.
Those estimated to be 13-17 years old will be limited to Teen User accounts.
Those estimated to be under-13 will lose their accounts altogether.
Those who open new accounts will have two weeks to submit to age estimation, and if they decline to do so, Meta is now required to treat them as a Teen User by default, even if they self-identify as being 18 and older. [P. 18, §II.A.10.b]
(b) Fourteen (14) days or more after creating a Meta SMP account, Meta SMP users who have not yet had their age assessed by an Age Assurance Method pursuant to the Age Assurance Framework shall be treated as Teen Users for the purposes of this Agreement regardless of their stated age, except that Meta SMP users with a stated age of 18 years old or older shall receive the protections described in Section II.A.10.a.ii.
What about people with existing accounts, who are well past that two-week period to submit to age estimation? Will they also be defaulted to Teen User status if they decline age estimation? It seems so—the AGs would likely not have accepted a settlement that did not require Meta to take action against existing teen users who choose to forgo the age assurance process. Perhaps Meta will use its existing store of information about its current users as a type of permitted proprietary age assurance process? Thus, perhaps, an adult user whose Facebook account is itself older than 18 will be assessed as being over-18? Or a user who is identified as the spouse of a user who has been age-assured? But Meta can only rely on a proprietary process if it meets the accuracy standards set out in the Settlement Agreement.
How accurate does the age assurance process need to be?
The Settlement sets maximum false positive for both commercially available and proprietary age assurance methods [Pages 12-13, §II.A.6]. Within two years, each shall be no more 10% for ages 16-17 and 3% for ages 13-15.
6. Age Assurance Standards. (a) U18 False Positive Rate Thresholds. (i) Any Commercially Available Age Assurance Methods deployed by Meta for any new and existing users in the Settling States shall meet or fall below the following U18 False Positive Rates excluding method circumvention within one year of the Effective Date: 10% for minors aged 16-17 and 3% for minors aged 13-15. (ii) Any Proprietary Age Assurance Methods deployed by Meta for any new and existing users in the Settling States shall meet or fall below the following U18 False Positive Rates excluding method circumvention within 1 and 2 years of the Effective Date: (A) Year 1: Within one year of the Effective Date: 14% for minors aged 16-17 and 7% for minors aged 13-15. (B) Year 2: Within two years of the Effective Date, 10% for minors aged 16-17 and 5% for minors aged 13-15.
Notably, there is no limit indicated in the error rate for false negatives, when the process wrongly identifies an adult as being under 18. The Settlement only requires Meta to maintain an appeal process for users whose age range is wrongly assessed. [P. 17, §II.A.9]
9. Appeals Process. Users claiming to have been mis-identified as minors must be offered a Clear and Conspicuous means to appeal the decision. Decisions on all user appeals must be made in a timely manner and communicated to the user along with a basis for the decision.
The Settlement generally shows little concern for those falsely placed in its Teen User category.
Meta must also employ measures to discourage age estimation circumvention, including placing limits on the number of attempts any user might make. [P. 16, §II.A.7] As part of this, Meta agrees to proactively monitor adult accounts to determine whether a user needs to undergo additional age estimation. [P. 16, §II.A.7.c] This is just one of the ways the Settlement embeds Meta’s active surveillance of its users for the next ten years (see below for more).
(c) Incorporating a proactive monitoring system that requires users to undergo an additional Age Assurance Method where a user is determined, including based on their conduct on Meta SMPs, to have likely circumvented the Age Assurance Method and is: (A) likely a Teen User after having been previously assessed as 18 or older; or (B) likely U13 after having been previously assessed as 13 or older. Users Meta determines are likely Teen Users after having been previously assessed as at least eighteen may choose not to undergo additional age assurance but then will be treated as Teen Users; and
Any age assurance process Meta uses must be tested annually.
Data minimization
The Settlement does have data minimization requirements for the data collected during the age assurance process. [§II.A.8] But there are numerous holes. The Settlement requires that all information obtained and retained as part of the age assurance processes thereafter be “immediately enqueued for deletion, after which it shall be deleted after a reasonable period of time.” But the Settlement defines a category of “Retainable Data” that may be retained for 90 days. This includes “metadata about the age assurance method used by the user information … where required to ensure the ongoing integrity of age assurance systems, including but not limited to the ability to identify circumvention and related patterns over time, and only for as long as it is required for those purposes.” The Settlement requires at that all data collected by Meta or its vendor be stored according to industry-standard data security measures,” a standard that unfortunately does not eliminate the risk of a data breach.
8. Data minimization and security. (a) Except as set forth herein, all data collected by Meta from users of Meta SMPs in the Settling States for the sole purpose of conducting age assurance, all data maintained from known U13s, and all data collected by a vendor for use in a Commercially Available Age Assurance Method shall be held for the minimum period required to determine a user’s age status and thereafter immediately enqueued for deletion, after which it shall be deleted in a reasonable period of time. Meta may retain (1) U13 data only to the extent required for purposes of developing, training, testing, and measuring the performance of the U13 Age Model (“U13 Data”), provided that any U13 data that constitutes Personally Identifiable Information as defined by 16 C.F.R. Part 312.2 will be protected using Meta’s highest data privacy and security standards, and (2) metadata about the age assurance method used by the user information (“Retainable Data”) only where required to ensure the ongoing integrity of age assurance systems, including but not limited to the ability to identify circumvention and related patterns over time, and only for as long as it is required for those purposes. For the avoidance of doubt, U13 Data cannot be used for purposes such as ads targeting and delivery, marketing, or algorithmic optimization efforts. Any U13 Data and Retainable Data shall be maintained at the coarsest viable granularity and cannot be used for any other purpose unless legally required. Any Retainable Data that is no longer required for the purposes set forth herein shall be deleted within 90 days. The terms above shall not pertain to the user’s stated date of birth, stated age, nor the outcome of the Age Assurance Method (e.g., “teen or adult” classification). (b) Data collected by Meta or a vendor, or transmitted by a vendor, must be collected and stored using industry-standard data security measures and as required by law, including encryption in transit and at rest. (c) The Parties agree to discuss in good faith potential modifications to this provision as necessary to permit Meta to improve the efficacy of its Age Assurance Framework while preserving the principles of data minimization and security set forth in this Section.
Restrictions For Teens (And Anyone Who Opts Out of Age-Gating)
Teen User Accounts are subject to time, feature, and content restrictions. These will be applied to these estimated to be 13-17 as well as any new user who declines to submit to the age assurance process and to existing users who decline to submit to age estimation and for whom Meta cannot ascertain that they are older than 17.
Time restrictions
Teen Users will be subject to the following time restrictions (§II.B). These measures seem to assume that most of teen’s social media use is frivolous and unserious (not that those are in and of themselves bad; the right to play is among young people’s human rights), ignoring the fact that teens use social media for school and personal research, conducting activism campaigns, and other endeavors that might naturally not fit within these time limits:
Night Access Mode – no access (except messaging) to Instagram and Facebook from Midnight to 6 AM, and no push notifications from 10 PM – 7 AM.
School mode – no push notifications from 8 AM – 3 PM Monday-Friday from Aug 15-June 15.
Daily cumulative time limit of 2 hours per day across Instagram and Facebook, resetting at midnight, excluding video and audio content at least 22 minutes long, absent artificial prolongation, defined by the Settlement as “longform content.”
“Productive pauses and notices” designed to “reduce or prevent excessive, mindless, or unintended teen usage.” This means that a teen’s usage will be monetarily paused after 60 and 90 minutes of daily cumulative use with notices sent every 15 minutes of continuous use. According to the Settlement, these productive pauses and notices will look like this:
To be clear, the ability to set time limits, blackout times, and scheduled pauses are all useful features that should be available and easy to implement for users of all ages. Such tools would have allowed teens, and all users, the ability to design their own safe experience, customized to their own needs, online. Such users controls would have recognized that teens have human rights, agency, and autonomy.
But that’s not what these restrictions are. They are not tools that give the teen users control. Rather, they are imposed, top-down, on teens and anyone else who declines to submit to Meta’s age assurance process.
Feature restrictions (§II.C-D)
Within four months of the effective date of the Settlement, Meta must offer teens an option for a non-personalized feed, which is defined as a feed of chronologically ordered posts from friends and follows. Teens will also be able to disable autoplay as part of an “optional protective settings” package. Each of these settings must be “viewable within three user gestures and clearly labeled, easy to notice, viewable without scrolling, and discoverable in an intuitive location within” the service.
Again, these would be useful user controls that should be offered to users of all ages.
By default, teens will not see the number of likes or other reactions to their posts.
Teens will also not have access to what the Settlement calls “Cosmetic Procedure Filters,” that is, “any digital filter or augmented reality effect that distorts, sculpts, redefines, or idealizes a user’s face in a way that cannot be achieved without cosmetic surgery or extreme make-up techniques.
X. “Cosmetic Procedure Filter” shall mean any digital filter or augmented reality (AR) effect that distorts, sculpts, redefines, or idealizes a user’s face in a way that cannot be achieved without cosmetic surgery or extreme makeup techniques. For the avoidance of doubt, “Cosmetic Procedure Filter” does not include: (1) Fantasy/Character Effects: Filters that change a user’s facial structure for the purpose of turning the user into a non-human or fictional character (e.g., an elf or cartoon) or an animal (e.g., a dragon or puppy); (2) Makeup/Smoothing Effects: Filters that smooth skin or alter appearance in ways that can be achieved by ordinary makeup techniques without altering the appearance of underlying physical facial structure or meaningfully changing skin tone; or (3) Parody and Exaggeration Effects: Effects that entertain users by distorting their appearance through parody, satire, or exaggerated forms (e.g., extreme visual distortions outside the scope of normal cosmetic procedures). To help operationalize this definition, the Settling States will provide Meta illustrative examples and guidance of AR effects that are Cosmetic Procedure Filters and AR effects that are not Cosmetic Procedure Filters in a letter to be sent within two (2) months of the Effective Date.
Meta has already had rules about cosmetic effects directed at teens since 2019. But the Settlement will give the states a major role in helping Meta identify what features are and are not Cosmetic Procedure Filters.
Content restrictions (P.1, §II.E, as defined by §I.C, E, F)
For content, Meta is basically pledging to continue its existing practices limiting Teen Users to age-appropriate content and accounts, to default Teen Users to age-appropriate experiences. This includes limiting access to accounts that “regularly share content that is inappropriate for teens” such as content from the following Meta community standards categories: Adult Nudity & Sexual Activity, Restricted Goods & Services, Suicide, Self-Harm or Eating Disorders.
C. “Age Appropriate Experiences” refers to content that is appropriate for Teen Users based on input from experts, parents, and teens. For the purposes of this Agreement, Age Appropriate Experiences shall mean content captured in Meta’s applicable Ages 13+ content setting, which is tied to policies inspired by movie ratings for ages 13+ and parent feedback. D. “Age Assurance Methods” shall have the meaning set forth in Section II. E. “Age Inappropriate Accounts” refers to accounts that regularly share content that is inappropriate for teens or that have account information that otherwise suggests the account is inappropriate for Teen Users. For the purposes of this Agreement, Age Inappropriate Accounts shall mean accounts that: (1) regularly share Age Inappropriate Content in the following categories: Adult Nudity & Sexual Activity, Restricted Goods & Services, Suicide, Self-Harm or Eating Disorders, or (2) have account names or profile photos or bios that suggest the account is otherwise inappropriate for minors, based on Meta’s policies for the following categories: Adult Nudity & Sexual Activity, Restricted Goods & Services, Suicide, Self-Harm or Eating Disorders. F. “Age Inappropriate Content” refers to content that is generally perceived by U.S. parents, youth experts, and teens as not being appropriate for Teen Users. For the purposes of this Agreement, Age Inappropriate Content shall mean content prohibited by Meta’s Community Standards concerning bullying and harassment; nudity and sexual activity; child sexual exploitation, abuse, and nudity; sexually explicit language; suicide, self-harm and eating disorders; graphic violence and incitements to violence; gambling; and restricted substances or goods (including illegal drug use), as well as policies specifically focused on protections for Teen Users, including those regarding high-risk viral challenges and risky stunts.
And even more worrisome, even though this is just Meta continuing its existing practices, the Settlement empowers the states to enforce its provisions. [P. 40, §IV.C.1.i; §VII.C] That means that over the next ten years, the duration of the Settlement, Meta will face the threat that a state attorney general will pursue legal action against it because it disagrees with how Meta interprets these categories of community standards, and pressure Meta to eliminate Teen User access to posts about sexuality and reproductive and sexual health. And Meta will now lack the hard-earnedFirst Amendment defenses to make its own curatorial decisions.
C. Notwithstanding anything in Sections VIII.A-B above, a Settling State may take any action, including but not limited to legal action to enforce compliance with the Agreement, without delay if the Settling State believes that a threat to the health or safety of the public requires immediate action.
The Parental Supervision Tradeoff
All of these Teen User restrictions can be modified – but only if the Teen User enrolls in the Parental Supervision that links their account to a parent’s or guardian’s account. Once their accounts are linked, Parents can modify the Teen User settings to make them less restrictive (they need the teen’s permission if they want to make them more restrictive).
And Parental Supervision comes with a huge privacy tradeoff. In exchange for designating someone as their Parent, the Parent gets a lot of information about the Teen’s use: the usernames of all of the teen’s connections, reports on how much time the Teen User spends on a Meta service, the time spent watching longform content, usernames of all those messaging with Teen User, and any evidence Meta has about suspected secondary accounts. The Supervising Parent also gets notices of the teen’s repeated searches related to suicide, self-harm and eating disorders. [P. 28, §II.G]
Parental Supervision 1. Meta agrees to provide Supervising Parents with information concerning the amount of time their Teen User is spending on Meta SMPs, including separately for time on the Meta SMP, time using the Meta SMP’s messaging features, time viewing Longform Content (to the extent excluded from the calculation of daily limits pursuant to Section II.B.3.a.i), and the usernames of the Teen User’s social connections and individuals messaging the Teen User, and usernames of any user reported by a Teen User. In addition, Instagram will notify Supervising Parents should their Teen User engage in repeated searches for terms related to suicide, self-harm, or eating disorders, and Facebook will notify Supervising Parents should their Teen User engage in repeated searches for terms related to suicide, self-harm, or eating disorders. 2. For Teen Users, Supervising Parents will be notified on a daily basis when the Teen User communicates directly with each adult user for the first time with the Teen User and shall provide a link to the adult user’s profile to provide information such as their stated hometown or city or mutual connections, to the extent the information is publicly available. 3. Upon a Teen User enrolling Parental Supervision, Meta SMPs shall prompt the Supervising Parent to review their Teen User’s settings to consider making updates to support how their Teen User spends their time on Meta SMPs. Meta shall not preselect, recommend, or encourage settings to Supervising Parents that are less restrictive than the default settings in this Agreement. 4. Meta SMPs will periodically suggest checkups for Supervising Parents to do with their Teen Users to evaluate their settings and usage. 5. Meta SMPs agree to continue to take steps designed to encourage enrollment in their Parental Supervision tools. 6. If a Teen User is enrolled in Parental Supervision, Meta shall notify the Supervising Parent if the Teen User creates or explicitly links a new secondary Meta SMP account within their Accounts Center or has been linked to a Soft Matched Account through Meta SMPs, including a link to the secondary account’s profile. Meta SMPs shall automatically apply the Supervising Parent’s approved time management settings in Section II.B and content restrictions to all explicitly linked Meta SMP accounts and all Supervised Accounts for a Teen User belonging to the same Meta SMP.
Unfortunately, Meta doesn’t have a great way to learn whether there exists a genuine parent-Teen User relationship. But it will try to get it right. [§II.G.8]
This may be workable for young people with healthy and safe relationships with their parent or guardian. But obviously not good at all for a Teen User lacking such a safe relationship.
More Surveillance, Not Less
Moreover, rather than pushing Meta away from the collection, analysis, and retention of user data, the Settlement requires Meta to do more of it for the next ten years. Several provisions of the Settlement require Meta to carefully track its users use of Instagram and Facebook for the purposes of determining whether its age assurance framework and Teen User restrictions are working as intended.
For example:
Meta promises to review detected U13 users’ friend networks to identify other possible U13 users. [§II.A.6.(b).(i).(D)]
Meta will incorporate a “proactive monitoring system” to identify possible Teen Users among those who were initially determined to be older than 17 by its age assurance process. [§II.7.(c)]
Meta pledges to utilize and improve its existing “soft matching models” that track signals such as device IDs, phone numbers, and email addresses” to identify duplicate accounts. [§II.B.6.(a)]
Meta will continue to monitor Teen Users’ activity to “regularly evaluate the prevalence of Teen User exposure to Age Inappropriate Content, and Teen User exposure to experiences that are not Age Appropriate Experiences,” [§II.E.3] and also to “regularly evaluate the prevalence of Teen User exposure to Harmful Experiences on Meta SMPs.” [§II.F.3]
Monitor Teen Users’ messaging to identify message threads with potentially suspicious accounts. [§II.F.4]
The Settlement also requires Meta to actively monitor use of its services so that it can supply data to the Independent Auditor that will now monitor its compliance with the Settlement. “The Independent Auditor is entitled to access the non-privileged information, personnel, systems, and records that are reasonably relevant and sufficient to evaluate Meta’s implementation of the Injunctive Relief Terms, including, but not limited to, access to raw data; aggregated data; information; internal documents and communications” plus information from its age assurance processes, data regarding Teen User responses to the Productive Pauses, data regarding its models for soft matching of secondary accounts, data regarding the prevalence of Teen User exposure to Age Inappropriate Content and Harmful Experiences, and to experiences that are not Age Appropriate Experiences. [§III.E]
Moreover, one of the chief threats of Meta’ surveillance is the honeypot of data it creates that may be accessible to governments for law enforcement and other investigations. Nowhere in the Settlement do the 52 attorneys general pledge to not try to access all of the data the Settlement requires Meta to collect and retain.
Meta Has To Pay The States — Establishing Norms Beyond Meta
The Settlement also includes annual payments from Meta to the states, apparently proportionate to the size of each state’s teen user base. Over the ten year-life of the Settlement, these annual payments will total over $11 billion plus the prospect of an additional $5 billion—if Meta competitors adopt the same measures.
This quirk of the Settlement incentivizes the States to pursue similar age assurance processes and at-least-as-restrictive teen user measures for Meta’s chief existing competitors for teen use, YouTube, TikTok, and Snap, and for any new service that may gain widespread teen use over the life of the Settlement Agreement. If the states are able to get Meta’s competitors to adopt the same measures, then the states will get an additional $5 billion in annual payments. That’s quite the incentive for the states to pursue litigation and regulatory measures against those companies. All of this will further entrench age assurance and age-gating as the norm across online services.
1. In the event the Contingent Monetary Payment Trigger has occurred in a Settling State, Meta shall be obligated to pay to such Settling State ten equal installments in the amount as set out in Exhibit B (each, a “Contingency Installment Payment”).
2. Following the date the Contingent Monetary Payment Trigger has occurred, the Contingency Installment Payments shall be made to the Settling State on January 15 in each subsequent calendar year of the Agreement Term as follows:
(a) At the next scheduled payment date, the Settling State shall be paid the Contingency Installment Payment for that payment date and all prior payment dates. (b) For each of the remaining payment dates, the Settling State will be paid the Contingency Installment Payment for that payment date.
3. If a Settling State fails to achieve the Contingent Monetary Payment Trigger during the Agreement Term, the Contingency Installment Payments shall be permanently forfeited by such Settling State and retained by Meta. For the avoidance of doubt, no Settling State will have an obligation to repay the Contingency Payment
The Settlement is thus a bad deal for all users of Facebook and Instagram. It normalizes age gating and age assurance for millions of internet users. It denies teens the tools to create their own safe experiences online and places their social media experience firmly under the control of either Meta or their parents. And rather than addressing Meta’s collection, analysis, and retention of data about teens’ use of Instagram and Facebook, it binds Meta to continued surveillance.
Earlier this year, one of the smartest internet rights people around, Heather Burns, suggested the “Darnella Test” regarding any kind of “kid safety” rule online. It’s named after Darnella Frazier. You might not recognize her name, but you’ve seen her work. She was the 17-year-old who was walking to the corner shop when she happened to see Derek Chauvin kneeling on George Floyd’s neck, in the process of murdering him. She got out her phone and filmed it, enabling the world to see that the initial statement from the Minneapolis police — the one headlined “Man Dies After Medical Incident During Police Interaction” — was bullshit.
But she was a teen. On social media. So Burns came up with the Darnella test, to explore whether any particular internet regulation would have prevented Darnella from documenting the murder of George Floyd:
For any young person who is just going to be walking along the street for a snack and ends up witnessing something that nobody should ever see because it should not be happening. For that young person whose only recourse, at that moment in time, is to document and report.
And as that future Darnella pulls out their phone to document the event:
Would they be allowed to have that phone, at all, under xyz regulation?
Would they be allowed to have that social media account, at all, under xyz regulation?
Would they be allowed to upload video, or livestream content, under xyz regulation?
Would the video, because of their age-verified account information, be instantly flagged and/or taken down for violent content?
Would the video, as urgent journalistic content in the public interest, be suppressed and censored based not on the content within it but on the age of the person who filmed it?
That’s your Darnella test. Run through it as if your life depends on it. Because it just might.
Thankfully, unlike the various outright bans of social media for teens, the newly announced Meta settlement with most state AGs does not actually ban kids from social media. But that doesn’t mean it passes the Darnella test. At the very least, the settlement will make it much harder for the next Darnella to document and distribute things that need to be documented and distributed.
Let’s start by running the settlement’s terms through the Darnella Test, question by question, to see what a future Darnella would actually be able to do. Especially since the goal of the settlement is that it become the industry norm across all of social media.
The first question doesn’t really apply here, since it has nothing to do with kids having phones directly, so that passes the Darnella test.
You could argue that the second question regarding whether or not the next Darnella would have a social media account also passes, since nothing in the settlement officially bars teens from having an account. But it certainly could impact them indirectly. The age assurance requirements make signing up more of a hassle, so the next Darnella might not bother creating an account at all. And if she does, that verification is precisely what marks her account as a teen account — which is what triggers everything that follows.
We can lump the next three questions together, because they’re all about the content itself. And here is where it seems clear that the settlement agreement fails the Darnella test. The “age-appropriate content restrictions” and the “content restrictions” for teen users would almost certainly create real problems for a teenager documenting a murder like George Floyd’s. A video of a police officer kneeling on a man’s neck until he stops moving is graphic violence by any classifier’s reckoning — and the entire point of a verified teen account is that graphic violence doesn’t stay on it.
And on that last item in the test — whether or not “urgent journalistic content in the public interest” would be suppressed — there are serious problems, even if the video somehow slipped past the content restrictions. The settlement imposes a default two-hour daily time limit on teen accounts — which means the answer might depend on how much time the next Darnella had already burned scrolling before she happened to walk past a murder in progress. Considering that George Floyd was murdered in the evening (around 8:30pm) there’s a good chance a teen user would have already used up their allotted time.
Yes, it’s possible that the next Darnella might have more time due to a parent or guardian bypassing the two hour restriction, but we don’t know that ahead of time. And a teenager watching a man die on the pavement in front of her does not have time to go find a parent and ask them to unlock the app.
Also, crime — and other things worth documenting — doesn’t happen only during your waking hours. The settlement includes “night mode” restrictions that say that teens will not be able to post content between midnight and 6am. If Darnella happened to witness Floyd’s murder past midnight, she might be out of luck. Or, at least, the police would have extra hours to lock in their false narrative.
And this is the problem with so much of the discourse regarding child safety online. It starts from the position that the internet is inherently unsafe for kids, and that the only remaining question is how thoroughly to block them from it.
But reality is more complicated than that. A kid with a phone and a social media account might also be the only thing standing between the official story and the actual truth. A teenager may be the one livestreaming a school shooting from a locked classroom. Or be a bystander filming ICE violating the rights of people across America. They might be the person documenting police violently attacking protestors.
Yet, under a framework built around “protecting teens from the dangerous internet,” such things will face multiple hurdles. A verified minor uploading graphic, unmoderated content in real time will violate all sorts of rules.
Run the actual Darnella video of Derek Chauvin kneeling on George Floyd against the “industry wide” standard Meta just paid $17 billion to create. It doesn’t pass. It’s graphic violence, filmed by a verified teen account late in the evening, perhaps after she’s used up all her allotted time. If it were after midnight she’d be barred entirely from posting. The next Darnella video might not exist. But at least 52 Attorneys General get to pretend they “protected kids.”
Most age verification laws tend to fail at their primary goal of barring kids from being online or from entering only specially designated zones, not to mention they pose a significant threat to everyone’s privacy. Some proponents of these age-based internet restrictions think they’ve found the silver bullet: Zero-Knowledge Proofs (ZKPs). We wrote about ZKP’s when they were first rolled out in the age verification context last year. However, more recent examples show our concerns weren’t just conjecture; ZKP-focused AV schemes are gameable, hackable, and not the cure-all some may claim.
ZKPs in Age Verification Would Only Centralize Power and Create More Harms
Before we jump into how these systems work, it must be said: creating a single point of failure for internet access contradicts the very idea of a free and open internet.
The mechanisms underlying ZKPs pose an existential threat to everyone’s digital rights, not just kids. The idea behind ZKPs is that you are issued a “token” that vouches for your age every time you log in, creating a constant link back to the entity that verified you. The issuer of the tokens these AV schemes rely on could track every time that credential is used, creating a dangerous trail of metadata on any user they wanted to target. The issuer itself could be pressured by authoritarian governments to remove a user’s access to a service, essentially removing that person’s access to the internet entirely. Without oversight of who has authority to implement and operate these systems, this approach centralizes critical internet infrastructure in the hands of very few actors.
How ZKPs Work
ZKPs are mathematically impressive cryptographic tools—but they weren’t developed with age verification in mind. Essentially, they let a computer quickly attest to the validity of a given question asked by another computer without divulging any underlying private data.
Computer A (such as the device operated by a person trying to access a website) is able to prove to Computer B (such as the server for the website that person is trying to access) that something is true without actually sharing the contents of that information itself. Computer A locks in a “commitment” to the information it needs to convey. Computer B, which wants to verify that information, generates mathematical “challenges” that can be answered correctly only if the information is true. Traditionally, this happens over many different “challenges” until there is no room for doubt that Computer A’s “commitment” is true.
Since that kind of lengthy back-and-forth process would drastically slow things down over the internet, there’s a shortened version of this exchange that’s “non-interactive.” In that case, the ZKP is verified instantly. The answer itself is hashed (mathematically converted into a fixed, shorter string of characters), and the resulting hash is theoretically unpredictable and tamper-resistant. This shortened version of the ZKP exchange is called “zk-SNARK,” which is the current preferred method for age verification.
In the ideal scenario, this means that ZKP’s are able to attest to a person’s status as an adult or a child without actually giving away any other private information about that person. In other words, only one entity would collect that private information, typically on the user’s device, instead of every website or app that needs the user’s age attested to. Unfortunately, recent real-world testing of these systems prove that ZKP’s aren’t the silver bullet that proponents of AV laws were hoping for.
EU’s AV Rollout Reveals How Broken It Is
By the end of 2026, the 27 states within the European Union are expected to have infrastructure in place to do age verification within a “mini-wallet” app that will live inside the EUDI (European Digital Identity) Wallet. This is being met with plenty of warranted criticism from digital rights experts. The “mini-wallet” version is already being rolled out, with promises that the ZKPs are in working order. But recent insights show that the ZKP features aren’t yet turned on except for the closed demo/prototype build (not the version of the app people are using “out of the box”), which the vast majority of everyday users can’t access.
Worse still, a security researcher found they could bypass the app’s system using a quickly built Chrome extension that tricked the app into repeatedly accepting the same “over-18” token. It did so without ever asking for fresh verification.
Over 400 security researchers signed an open letter stating that age assurance checkpoints, even if implemented with privacy in mind, would cause more harm than good. A primary focus of their concern, which we share, is the fact that a centralized identity verification system creates a single point of failure that is extremely vulnerable to both cyberattack and authoritarian overreach.
Once the “mini-wallet” version of this is fully integrated into the EUDI Wallet, it will replicate these same failures, perhaps more, but at a much larger scale. At that point, the failures will involve many more pieces of sensitive information that the EUDI Wallet contains: passports, driver’s licenses, travel information, financial information, to name a few.
ZKP’s Aren’t The Magic Bullet
As we’ve said time and time again, no method of online age verification is privacy-protective, fully accurate, and capable of guaranteeing universal coverage without introducing severe security risks.
Lawmakers concerned about the privacy failures of age verification mandates must understand that ZKPs are not a magic bullet. They do not solve the age verification paradox; they simply push the burden of trust down the road, relying on technical ignorance and magical thinking about how the internet actually functions.
Even as the wider EU was grinding through a long, drawn-out process to figure out which regulatory levers to pull on kids’ safety online, France decided to YOLO it earlier this summer by jumping at the chance to ban all social media for kids under 15. This kind of thing has become popular with out of touch adults in the grips of an ongoing moral panic, since Australia led the way after gambling companies pushed the ban as an alternative to banning gambling ads. Australia’s ban isn’t doing very well, with the majority of kids figuring out how to work around it, and those being left out being the most marginalized and in need of community.
And yet, countries around the globe have all suddenly decided — some based on reading Jonathan Haidt’s badly reasoned book — that they must do this too.
France was the first in the EU, with President Emmanuel Macron gleefully treating France’s willingness to rush in with little thought or understanding as a selling point:
“France is leading the way in Europe in protecting our children and our teenagers,” Macron said. “We will keep on going.”
He wanted the ban to be implemented in mere months, just as kids returned to school.
But that’s all run into a bit of a stumbling block known as the French Constitutional Council, which has said the ban is an unconstitutional attack on kids’ rights to free expression and communication. The Council also flagged a second problem: you can’t enforce an under-15 ban without making every user, adults included, prove their age. The law demanded exactly that, without defining a single condition, limit, or technical standard for how that verification would work.
A court in France on Friday shot downa bill seeking to ban access to social media for under-15s from September — a major blow to President Emmanuel Macron that raises fundamental questions about efforts to protect kids on the internet.
The Constitutional Council, which reviews the constitutionality of French legislation, said the restrictions in the bill disproportionately infringe on minors’ right to freedom of expression and communication.
Reading through the Google translation of the actual ruling, it’s pretty short and to the point. Similar to how the First Amendment requires any restriction on speech to be narrowly tailored to a specific government interest, here the Council says the ban is way too broad and not based on any specific, narrowly defined harm, though it admits that the aims of protecting children are certainly legitimate:
Furthermore, while the established prohibition does not apply to online encyclopedias, educational or scientific directories, or platforms for developing and sharing free software or open-source educational digital projects, the exceptions provided for in the contested provisions remain limited. In particular, these exemptions do not cover collaborative services for sharing leisure, information, or mutual aid content, online communication applications, or online games with strong collaborative and social features, nor do they cover online social networks which, while not inherently educational, are created in connection with educational activities.
Thus, the prohibition established is likely to apply to online communication services whose risks to the health and safety of minors, relating in particular to their content or mode of operation, are not established.
In short, the law goes way too far in issuing a blanket ban of all children, no matter what the circumstances. If you have a legitimate, well-defined problem, come up with a narrowly tailored solution. The French government rushed this one through with little concern for things like that.
It also leaves little room for parents to decide for their own kids what’s appropriate:
… neither the contested provisions nor any other provision sets out the conditions under which the holders of parental authority or the legal representative of the minor, duly informed of the potential risks and safeguards presented by the services concerned, may, in the child’s best interests and in the exercise of their duties under the law, decide to lift the prohibition, limit its scope or authorise access to certain services.
Thus, the prohibition established does not give rise to any particular assessment of the risk to the minor, taking into account in particular his age, his degree of maturity, his family situation as well as the nature of the service concerned.
Also, the age verification attack on privacy is important to recognize:
By prohibiting access for any minor under fifteen years of age to certain online services, the contested provisions imply, in themselves, that any person, even an adult, must prove their age before accessing them.
The Council further notes that the law makes no real effort to figure out how one might implement age verification in a manner that protects the privacy rights of adults.
Of course, having hitched his own legacy to this thing, there’s no way Macron lets it go quietly:
In a statement late Friday, the French presidential office said the government would not be giving up on the bill. It has set a new target date for spring 2027, which coincides with when Macron will leave office.
The statement said Macron “has instructed the Prime Minister to work, as quickly as possible, on a legally sound draft that takes into account” the court’s decision. The ruling hinted at what would make the age restriction align with fundamental rights: giving parents more flexibility.
Politico also spoke to Peter Craddock, a Brussels-based attorney who works on social media regulation, who notes that any other EU country attempting a similar blanket ban is likely to run into exactly the same wall:
“The reasoning is actually equally relevant internationally, throughout the EU, because this fundamental freedom is not specific to France,” he said.
Which is a useful reminder that the freedom of expression problem here isn’t a quirk of French constitutional law. It’s baked into the whole approach — and no amount of “but it’s for the children” framing makes it go away. That’s even more true of the problems with age verification requiring the scanning of everyone’s ID, which is an even touchier subject in large parts of the EU than elsewhere.
Really, though, the bigger, more important message here should be to slow down. What’s incredible is that for all of the political and media class whining that social media is some rogue experiment on our children, none of them seem to consider that abruptly trying to block all social media from kids is just as much an experiment, and one that might have equally damaging effects.
Why not wait and see how the Australian ban actually works in practice? The early results are a mess. I get that Macron and other politicians want headlines and a legacy to point at, but it would be nice if they actually followed what the research shows and looked at how the early experiments of these bans have worked out.
So what France produced here was a total rush job that sacrificed the expression rights of every teenager in the country, the ability of parents to make their own judgment calls regarding their own kids’ access to information and — as a cherry on top — the privacy of every adult who would now need to prove their age at the door to the internet. Thankfully, the Constitutional Council caught all three.
Earlier this year, following the initial verdict in the state of New Mexico’s case against Meta (arguing that Meta caused harm to children by doing things like encrypting messages), we noted that even if you believe that Meta is a terrible company (as I do), that Mark Zuckerberg cannot be trusted (ditto), and that the company has prioritized profits over safety (yup), we should be seriously concerned about the verdict. Among the many problems with the verdict is that it’s a judge and jury taking on the role of determining what they feel is how internet apps can and should work, despite having no knowledge or expertise regarding how internet products can and should work — or how trust and safety tradeoffs actually work.
The second part of that trial, focused on remedies beyond the monetary judgment the jury already awarded the state, concluded last week. Almost all of the headlines are talking about the additional $567 million the judge ordered the company to pay above and beyond the $375 million in civil penalties the jury already awarded. Those funds are supposed to be used to help pay for the supposed “harms” of Meta’s properties to New Mexico kids. But that is probably the least interesting part of the ruling.
What’s way scarier is that the judge then makes a bunch of design decisions, telling Meta how its products need to work. Admittedly, many of the changes are ones Meta itself suggested (some of which it has already made). But this is exactly the kind of thing that I was worried about when the first verdict came down. Once we’re in a world where a judge gets to make product decisions, bad things are going to happen. Judges are not product designers. Judges are not trust & safety experts. Judges are not child safety experts. They do not understand how all these things work together. They do not understand the tradeoffs of their decisions.
Judges simply shouldn’t be in the business of determining the proper user interface for software. Indeed, I’d deem it to be an example of compelled speech and (in cases of banning certain features) suppression of speech.
In the ruling, the judge now says that Meta will need to limit the amount of time children can spend on the app (which means mandating age verification of every user), that the number of “likes” on photos on kids’ accounts must be hidden, and then some compelled “warnings” about the supposed harms of social media. All of this is problematic.
Again, even if you hate Facebook and think it’s harmful and you trust this judge, do you trust all judges out there to get this right? Do you think that some of the Trump appointed judges should be able to decide what features certain apps should have? We’ve already seen some judges who think that any trust & safety/content moderation operation is harmful. And we’ve seen judges on the flip side who think that not taking down certain speech is equally harmful. Letting individual judges determine how products are designed is a disaster waiting to happen.
As we’ve discussed for years, there is scant evidence that social media (and Facebook’s apps in particular, as they’re the most studied) are inherently harmful to kids. Many, many, manyresearchers have tried to find such evidence and tend to come up empty. It would be one thing for the judge in this case, Bryan Biedscheid, to just ignore all of that, but he doesn’t. He more or less admits it, highlighting the various claims from both sides in the case… and then just declares which side he thinks is more credible. But a different judge might find otherwise. Which is a big part of the problem here.
If all the science agreed, this would, perhaps, be an easier call. But the vast majority of the science disagrees with Biedscheid’s feelings here.
He starts out by citing the Surgeon General’s report from 2023, but reads it to say a lot more than it actually does:
In 2023, the U.S. Surgeon General issued a public advisory titled “Social Media and Youth Mental Health” (“2023 Advisory”) which “calls attention to the growing concerns about the effects of social media on youth mental health.” [Pl. Ex. 03193 at 3] As the 2023 Advisory notes, “[a] Surgeon General’s Advisory is a public statement that calls the American people’s attention to an urgent public health issue,” and such advisories “are reserved for significant public health challenges that require the nation’s immediate awareness and action.” [Pl. Ex. 03193 at 3] The 2023 Advisory states that “[u]p to 95% of youth ages 13-17 report using a social media platform, with more than a third saying they use social media ‘almost constantly.’” [Pl. Ex. 03193 at 4] The 2023 Advisory also notes that “nearly 40% of children ages 8-12 use social media.” [Pl. Ex. 03193 at 4]
Importantly, the 2023 Advisory reports that features designed to maximize engagement can harm children by encouraging problematic use and behaviors. The 2023 Advisory provides that:
[e]xcessive and problematic use of social media can harm children and adolescents by disrupting important healthy behaviors. Social media platforms are often designed to maximize user engagement, which has the potential to encourage excessive use and behavioral dysregulation. Push notifications, autoplay, infinite scroll, quantifying and displaying popularity (i.e., “likes”), and algorithms that leverage user data to serve content recommendations are some examples of these features that maximize engagement.
But that proves nothing. Note even the hedging language here. It says that excessive or problematic use can harm children… but doesn’t say how often that occurs. It says that those features have “the potential to encourage excessive use” but makes no findings on whether they actually do or how frequently they do.
Indeed, as we keep pointing out, where the evidence actually takes us is that a very small percentage of kids cannot handle unsupervised, unlimited social media. Most of the studies appear to put it at less than 5%. And, much of the research seems to suggest that any causal connection is in the other direction. That kids who are not getting the mental health support they need then turn to social media and use it excessively, rather than the social media causing the problems.
Also, the judge completely leaves out that the same Surgeon General report talks about how incredibly helpful social media is for many kids. He does mention elsewhere other studies showing that social media has benefits… but then forgets all about it.
The evidence in this case demonstrates that social media has many benefits, such as helping people connect with friends and family. [See, e.g., 2-27-26 Tr. 7225:11-14 (Cain); 2-19-26 Tr. 4904:1-11 (Coyle); Def. Ex. 01045 at 93] It also helps members of marginalized communities form and build online communities based on connections and shared interests, especially when members of those communities might not have access to supportive or even safe communities offline. [2-12-26 Tr. 2452:22-25 (Boyle); 3-11-26 Tr. 10882:22-10883:16 (Otaru)] It is also a source of news, educational content, and entertainment. [Def. Ex. 01045 at 93] Meta’s services in particular help small businesses grow by allowing them to reach more customers.
In other words, the issue is not about social media inherently. And any solution across the board is a terrible idea. Instead, we should be making efforts to identify which kids are actually at risk and helping them. Not just universally declaring which features are good and which are bad.
Unfortunately, that’s what the judge does here, effectively appointing himself the product manager for Meta’s social media products and saying that despite the conflicting evidence, he’s convinced that Meta’s products are designed to be harmful to children:
Based on the evidence in the record, the Court finds that Meta implemented platform features that were designed to optimize engagement, and that these features were and are harmful to teenagers. [See, e.g. 2-17-26 Tr. 3573:15-17, 3581:15-3582:16 (Narayanan) (outlining how Meta’s algorithm uses engagement-maximizing recommendation algorithms); 2-23-26 Tr. 5468:22-5470:15 (Lembke) (describing how children are “uniquely vulnerable” to harm on social media because their brains are still developing)]
But… optimizing engagement is what every cultural product does. Many novels I read end each chapter on a cliffhanger, and sometimes that gets me to stay up late at night as I need to know what happens next. That’s harmful to my sleep. And it’s a design decision from the author/publisher to end chapters like that. But I think we all recognize that it would be a massive First Amendment problem if we told publishers that they couldn’t end chapters on cliffhangers, saying that they “optimize engagement” and therefore are “harmful.”
Basically, the court decides that the experts the state put on the stand were more credible than the experts Meta put on the stand. But just the fact that there’s so little agreement about what the science actually says here, and no study that has shown causality, should give the judge pause. Instead, because one witness claimed a causal relationship, he accepts that as fact.
It would be nice if the judge showed a bit of humility here, but instead seems to assume that based on a few witnesses at the trial he can redesign Facebook and Instagram.
The judge, who had previously rejected Meta’s Section 230 defense, admits that he can’t do anything regarding actual content on the app, because that would likely violate both 230 and the First Amendment:
This decision does not seek to close or demolish Meta’s platforms, enjoin specific advertising or content (which the Court is also mindful is protected by the First Amendment and 47 U.S.C. Section 230 of the federal Communications Decency Act (CDA)), or require Meta to cure causally disconnected harms. Rather, this decision seeks to address existing harms created by Meta’s platforms and to prevent future harm to children and future burdens on New Mexico, in general, that would otherwise be caused by Meta’s platforms. It is this Court’s conclusion that Meta is a cause of and has substantially contributed to a public nuisance in New Mexico, and is required to abate that public nuisance to the extent of its contributions.
But Judge Biedscheid still appoints himself the new product manager for safety for Meta’s products for other features. He establishes that Meta’s social media product features represent a “public nuisance” to kids in New Mexico, and they must abate the harm of that public nuisance. First, it tells Meta that it needs to implement age verification, though it admits that it’s somewhat limited by federal law (COPPA) in how much it can require:
Age verification is the key to making Meta’s platforms safe for adolescents because there must be substantial certainty about whether a user is: (a) over 13 years of age, thus old enough to use the platforms; and, (b) over 18 years of age, so that adolescent protective restrictions on use should be removed and restrictions on the ability to connect with adolescents must be put in place. Without substantial certainty, there will be adults communicating with adolescents by claiming a false younger age and tweens communicating with teens and adults by claiming a false older age.
But… most experts don’t think that’s true. Indeed, many experts have made it clear that age verification introduces all sorts of new risks and dangers. The letter linked there (from 438 experts) was even brought up during this phase of the trial, but was dismissed as being by experts who were out of touch by New Mexico’s experts (and, apparently, the judge).
And it’s this kind of thing that feels so problematic about this. The science isn’t even remotely settled here, and there are credible experts warning that these solutions will actually do way more harm than good for children’s safety. Shouldn’t that be important too?
What if these changes demanded by the judge actually do more harm?
Then what? Do we get to sue the state of New Mexico for being a public nuisance? Or sue Judge Biedscheid? Nowhere in this order is any sort of humility or acknowledgement that he has no idea the actual impact of these changes. There is no plan to go back and look and check to see if they are working. Or if they’re causing more harm. If Meta makes these changes and the rate of youth depression or suicide increases next year, then what? Whom do we get to sue? Who “abates” that public nuisance?
And then there are other features that Product Manager Biedscheid decides Meta can no longer offer to kids (as determined by the age verification tools he’s requiring them to use):
Meta shall eliminate push notifications on its platforms for known or estimated accounts belonging to users under 18 years of age: (a) from 10:00 PM to 7:00 AM (i.e., hours when the vast majority of children are sleeping or should be asleep) on all days; and, (b) from 8:00 AM to 3:00 PM (i.e., typical school hours) during the academic year, excluding weekends. Notwithstanding the foregoing, Meta may make exclusions to this rule for messaging from connected users and for urgent, targeted messages, such as for security or hazard alerts
Meta shall hide, as a default setting for all accounts where the user is under 18 years of age, all “like counts” that Meta appends to content. Meta may only allow an override of the default setting when a parent or guardian gives the user under 18 years of age their permission to change the default setting.
Meta shall implement a mandatory usage time limit for accounts belonging to users under 18 years of age. Meta shall restrict the usage of all such users to not more than 90 hours of use per month cumulatively across Facebook and Instagram.
Do we know if any of this will help kids? ¯\_(ツ)_/¯
Will the court go back and revisit this if this magically doesn’t help kids mental health struggles? ¯\_(ツ)_/¯
Meta shall provide information screens on Facebook and Instagram to be displayed to all new users under 18 years of age once a day that explain one or more of the following: (a) an aspect of safe platform use best practices; (b) available tools to address inappropriate content and behavior; (c) the concept of rabbit holing and ways to reset the algorithm to address the issue; and, (d) tools to avoid problematic use, such as nudges, self-imposed limits and other tools. These screens shall be displayed for the first 30 days that a new user under 18 years of age joins a platform, similar to Meta’s presentation of new users under 18 years of age with Pristine Pool content. The screens shall be submitted to the State for its review, possible edits, and approval.
Meta shall fund, design, and implement an educational campaign in New Mexico, prepared in collaboration with and with the approval of the State, that publicizes: (a) risks associated with platform use; (b) safety tools and best practices to address those risks; (c) parental controls; (d) problematic content and behavior reporting tools; (d) risks associated with online bullying and features that help victims address it; and, (e) reporting mechanisms for accounts belonging to users under 13 years of age. This campaign shall also provide schools with ready-made materials, also prepared in collaboration with and with the approval of the State, that set forth information on these topics to be distributed as desired by the State.
It’s entirely possible that those are good ideas. But we should all be worried about judges ordering internet companies what they must tell their users. Remember, we’ve seen other attempts to do this — such as the law in Texas that tried to force adult content sites to warn people that porn was dangerous. And those were thrown out as unconstitutional.
But how do you distinguish these warnings from the anti-porn disclosure warnings pushed by the far right? The point is that the state isn’t supposed to be in the business of requiring warnings on inherently speech related products. Especially when there is no agreed upon scientific basis for the claims.
About the only good thing in the ruling is that the judge neglects to take the state up on its worst suggestion: banning end-to-end encryption in WhatsApp or Facebook. But even that is only partially good, because he orders Meta not to turn end-to-end encryption back on for Instagram. As you’ll recall, back in May, Instagram turned off end-to-end encryption, claiming that no one was using it. Privacy advocates have asked the company to turn it back on, but if this ruling stands, that won’t be allowed. The judge’s rationale is basically that (1) WhatsApp doesn’t have the other “addictive” features he dislikes and (2) kids don’t use Facebook anyway, so only Instagram is the problem:
The Court does not adopt Plaintiff’s requests for a prohibition on end-to-end encryption (“E2EE”) for the following reasons. Meta has already ceased offering E2EE on its Instagram platform, and the Court orders that such cessation of E2EE on Instagram remain in place during the Abatement Period. Regarding Facebook, the Court does not agree with ending E2EE on this platform. Facebook has few adolescent users in New Mexico, and the Court only has jurisdiction over New Mexico accounts. Therefore, in light of the benefits E2EE may offer in many other markets outside of New Mexico, and the limited use of E2EE by adolescents on Facebook in New Mexico, the Court does not impose a prohibition on E2EE in relation to Facebook.
But, again, on what scientific basis is this decision made? It all feels like vibes.
It’s also weird because a few paragraphs later, the judge admits that the benefits of end-to-end encryption outweigh the harms… but only on WhatsApp? On Instagram it’s the opposite? Why?
To the extent that WhatsApp’s E2EE poses a risk to adolescents, that risk is outweighed by the benefits that E2EE offers to address privacy concerns of New Mexicans and other populations.
He also admits that while he might like to force Meta to turn off autoplay, infinite scroll, and content recommendations, he (correctly) worries that those (since they’re so directly tied to third party content and editorial decision making) could implicate both Section 230 and the First Amendment:
Of the above-mentioned design features, autoplay, infinite scroll and algorithmic content recommendations are most closely tied to content presentation because they directly impact the manner in which users are presented with third-party postings and advertising. As a result, autoplay, infinite scroll, and algorithmic content are features with clear Section 230 and First Amendment implications.
The evidence and argument at trial also showed that autoplay, infinite scroll, and algorithmic content recommendations are used widely in the industry. Therefore, in light of the broader market in which Meta operates and given the absence of Meta’s competitors in this litigation, restrictions imposed on Meta’s offering of the aforementioned features could harm the viability of Meta and its platforms.
In contrast, push notifications and “like counts” are least connected with platform content. Indeed, push notifications occur in many instances even when the user is not directly interacting with the “pushing” application on their device. Additionally, “like counts” are merely a feature created and offered by Meta to principally track and motivate user feedback, all without altering the underlying published content.
As with Age Assurance, while the Court agrees with the State that autoplay, infinite scroll, “like counts,” and algorithmic recommendations combine to facilitate addictive or problematic behaviors in adolescent users, the Court does not see an equitable abatement method: (a) that impacts only Meta, rather than imposing restrictions industry-wide; and, (b) that respects the protections of the First Amendment and Section 230.
The simple fact is that all of this is one judge issuing orders regarding how a massive company with billions of users needs to be designed. And, yes, he’s trying to keep the platform safe but he’s making these decisions based on his feelings about what will work, and he’s no expert. And there’s nothing in there considering what happens if these changes actually make things worse, as they absolutely could do.
Within Meta, I’m quite sure they measure every little change. They check to see what works and what doesn’t. And when things don’t work, they adjust. Frequently. They have KPIs to meet and regular reviews. If something goes wrong they roll stuff back re-evaluate. They run A/B tests. They explore each decision, and they have to continually justify the decisions they make.
As their new product manager, Judge Biedscheid has none of that. He gets to declare from on high what changes Meta must make to their product, and… that’s it. The design decisions are his. But the responsibility isn’t. If teen mental health in New Mexico doesn’t improve — or if it gets worse — there’s no plan to review. There’s no rolling it back. There’s no A/B testing. Judge Biedscheid is off on another case.
Recent reports have raised alarm about the use of PatronScan, an ID-checking and face-scanning system, at multiple LGBTQ+ bars in San Francisco’s Castro neighborhood. Much of the attention has focused on reports that the system photographs patrons as they enter venues and questions about whether those images are used for facial recognition.
A broader privacy concern also deserves scrutiny. For years, PatronScan has marketed itself not just as an ID-verification tool, but as a system that allows bars and clubs to identify patrons, keep records about them, and share information across venues. As one news article published in 2019 documented, PatronScan built a network that allowed participating bars to flag patrons and share information about them with other establishments.
And in California, it’s not at all clear how PatronScan’s business model of scanning IDs and sharing the information from those scans with other bars comports with the law. California’s ID privacy law, which was amended in 2018 to add ID “scans,” states that no businesses shall “retain or use” any information from a scanned ID card except for limited purposes such as to verify age, comply with a legal requirement, or prevent fraud.
Californians should be deeply concerned about businesses that collect information from government-issued IDs and use it to build databases about where people go, whom they associate with, and whether they should be allowed into other public gathering places. That concern is especially strong in LGBTQ+ spaces, which have long served as refuges for people to go without being tracked, monitored, or put on lists.
We reached out to Patronscan with questions regarding their practices and their views on California ID law. They referred us to their published FAQ question “Is Patronscan privacy compliant in California?” which claims that the use of Patronscan kiosks is legal in California. They also said “Patronscan does not do facial recognition in North America, or any kind of automated analysis of the ID or the live photo image.”
The California Legislature Has Investigated PatronScan’s Business Model
In 2018, the California Legislature published bill analyses (on that year’s AB 2769) that went into detail about PatronScan’s business. Reviewing PatronScan’s own materials, the California Senate Judiciary Committee found that the company had collected and retained information on 561,087 customers in Sacramento alone during the first five months of 2018—a remarkable figure for a city whose population had only recently topped 500,000.
Lawmakers also found that at that time, PatronScan retained information for at least 90 days or longer in some cases, shared information among participating bars, and maintained bans that lasted an average of more than 19 years. A PatronScan “Public Safety Report” used 10,000 scans collected on a single day to report on “where customers live, how far they have traveled, and how many different venues the customers patronized.”
This was not simply checking IDs at the door. PatronScan was building a database.
An immigrants’ rights group, the Coalition for Human Immigrant Rights (CHIRLA), wrote about its concern at the time with these growing ID databases, saying that “placing individuals on a database that labels them a “threat to public safety” has “significant immigration consequences that could lead to deportation, revoking of current status, or denial of future immigration relief.”
Today, PatronScan states that it retains personal information about all customers for 21 days, and about flagged customers for up to five years. This includes the customer’s name, date of birth, photograph, gender, and zip code. It also includes the dates and times that the customer entered particular bars. Such databases are a grave privacy threat. Personal data is routinely stolen by thieves, misused by a company’s employees, seized by government agencies, and diverted to new purposes by a company’s executives.
California Law Still Bans ID-Scan Databases, And Bars Should Follow That Law
In 2018, California lawmakers closed what they viewed as a loophole. Existing law already prohibited businesses from retaining or using information obtained when they “swiped” a driver’s license, except for the narrow purposes of legal requirements (like a judicial warrant) or “preventing fraud, abuse, or material misrepresentation.”
After reviewing companies like PatronScan, the Legislature amended the law to make clear that the same restrictions that apply to businesses that “swipe” ID cards also apply when those IDs are “scanned.” PatronScan opposed that change, arguing it wanted to preserve the ability to share information among bars so participating venues could decide whether to admit patrons.
The bill became law anyway. Yet PatronScan continues to market and sell a system that apparently retains information from scanned IDs, and allows participating venues to flag patrons and share information across its network.
At a minimum, that raises serious questions about how those practices fit with California’s existing ID privacy law. Bar and nightlife venue owners who utilize PatronScan should think twice about its effects on their customers, and consider going back to standard, visual ID checks. These physical checks have been effective at keeping underage patrons out of 21-and-over venues for decades, and don’t present the serious privacy dangers of creating a private database of bar patrons.
For venues serving vulnerable communities like immigrants or the LGBTQ+ community, the stakes of using this technology are even higher. It’s disappointing and alarming to see some of California’s more well-known LGBTQ+ nightlife spots instead lining up as PatronScan’s early adopters. A venue cannot claim to be a safe space while feeding its patrons’ data to a third party database. These businesses should reject PatronScan, return to the standard ID checks that every other bar has been able to utilize, and prove to their customers that their privacy and security still matters.
The Senate Commerce Committee is set to consider S. 737, the SCREEN Act, a sweeping age-verification bill that would require online services to verify users’ ages before they can access any sexually explicit content. If this bill passes, it will force millions of adult internet users to give up their anonymity, privacy, and security before they access lawful speech.
Unlike many state-age verification laws—which have been harmful in their own right—the SCREEN Act has no requirement that a significant portion of the website consist of sexually explicit content that is harmful to minors. The bill requires nearly any service hosting even a single piece of sexually explicit content to verify the ages of its users. The result is that the bill would apply not only to adult content sites like PornHub or OnlyFans, but also streaming services like Netflix, and social media platforms like Reddit, Discord, or Bluesky, if they host any adult content.
The SCREEN Act does not merely require users to attest they are adults. It specifically states that “requiring a user to confirm that the user is not a minor shall not be sufficient.” In practice, that means platforms would have to verify users’ ages using methods tied to their real identities. Providing proof of age online is dramatically different, and far more invasive, than showing your ID at the door to a bartender or bouncer. In the physical world, the bouncer at the door looks at your ID card, confirms you’re old enough, and gives it back to you. Under the SCREEN Act, the “bouncer” will be a digital age-verification service that captures your personal information and saves it to a database for an unspecified amount of time.
The consequences of the bill won’t be limited to minors. If websites and apps are expected to reliably identify teenagers, adults will be asked to prove they are adults.
Even worse, the SCREEN Act is a privacy and data security nightmare. One provision of the bill requires services to take reasonable steps to protect the data collected and to not maintain for longer than is necessary. But these are terribly weak protections that impose no meaningful collection, use, or retention limits on services collecting people’s private information.
In other words, the third parties tasked with verifying a user’s age on a platform could sweep up a lot of personal info they don’t actually need and then could use that information for any number of purposes, so long as they deem their actions reasonable. Companies would then be allowed to keep the information users have been compelled to turn over for as long as possible, raising security and privacy issues along the way.
The SCREEN Act Attacks Your Right To Use VPNs
The SCREEN Act also targets virtual private network (VPN) users and providers. The bill requires covered websites to verify users’ ages based on their IP addresses unless the service can determine that the user is outside the United States, and specifically requires age verification on traffic coming from known VPN addresses. In practice, this discourages the use of VPNs and proxy servers, which millions of people rely on for legitimate purposes such as protecting personal privacy, securing public Wi-Fi connections, safeguarding journalists and activists, and preventing data tracking.
VPNs mask your real location by routing your internet traffic through a server somewhere else. When you visit a website through a VPN, that website only sees the VPN server’s IP address, not your actual location. It’s like sending a letter through a P.O. box so the recipient doesn’t know where you really live. VPNs are a privacy and security tool used by millions of internet users every day, and their use should not be treated as suspect. It is particularly galling that the SCREEN Act forces users who intentionally take steps to protect their privacy to identify themselves.
The SCREEN Act creates onerous age-verification rules that will block adults from accessing lawful speech, curtail their ability to be anonymous, and jeopardize the data security and privacy of all internet users.
We — and plenty of others — have been warning that the global rush to mandate age verification wouldn’t stop at “let’s make sure kids can’t see porn” or even just “keep kids off of social media.” It would inevitably expand into treating anonymity and privacy tools themselves as the enemy. Australia is now proving that in real time: Its eSafety regulator has gone from checking whether porn sites gate their content to treating VPN use — one of the best tools people have for protecting their privacy online — as a compliance problem to be stamped out.
The correct term for age verification as it is implemented today is therefore identity verification. Given today’s internet infrastructure, it is unreasonable to assume that this information will not be shared through commercial agreements or with governments.
The consequence of introducing identity verification is therefore that freedom of information is restricted (you can no longer visit regulated websites anonymously) and that you can no longer post anonymously on social media. You cannot be certain that your criticism of the government will not be followed up by the authorities. You can no longer start a digital initiative on a social media platform aimed at gathering people to criticize an authority without facing a significant risk of consequences. Depending on the country you live in, this could even endanger your life. In its current form, social media identity verification removes important tools for activists in countries where criticizing those in power is dangerous.
Freedom of expression is threatened not only in a direct sense (you post something and then the police knock on your door), identity verification also creates a chilling effect. It becomes a cornerstone of censorship machinery in the sense that people begin to self-censor if they know that expressing opinions may have personal consequences. This is also something that changes over time. What is considered acceptable to post online is determined by whoever currently holds power. Different sides of politics often have different views on what constitutes harmful content. Just because what you post today is not considered inappropriate does not mean it will remain acceptable in the future.
Some can argue that they’re biased since they’re in the business of selling VPN service, though arguably, more age verification laws increase demand for VPNs. But, the reality is that as age verification laws spread, so too do the attacks on VPNs and the ridiculous and dangerous threats to somehow outlaw their usage.
The latest is in Australia, where their teen social media ban has been an abject failure. Have no fear, however, they’re going to just start targeting VPN usage. Of course, they’re not framing it as a response to the failure of their social media ban, but rather a response to adult content websites’ age verification being beaten by people using VPNs, because it’s always easier to start your attacks on privacy, security, and anonymity by blaming a more marginalized industry like adult content:
Nine in 10 of the most visited adult sites used by Australians now have age checks for users, according to the online safety regulator, but eSafety has said it will assess whether those sites are allowing users to bypass restrictions with virtual private networks (VPNs)….
But, of course, it’s not just about adult content. They’ll go after VPN usage for social media as well:
Similar to the expectations of the social media companies for the under-16s ban, eSafety said it was expected under the codes that sites “must take reasonable steps” to prevent workarounds like VPNs, and eSafety “will look at this when considering compliance”.
The sheer irony of an agency named “eSafety” claiming that VPN use was a “workaround” that must be blocked? VPNs provide way more safety than anything that the “eSafety” Commission has done regarding internet usage.
Age verification is surveillance. Full stop. And it’s increasingly being closely tied to law enforcement and governments. Tech policy expert Heather Burns recently pointed out that age verification providers were literally reporting people to law enforcement for the crime of… using an alternative OS. As she notes:
age verification providers now hold themselves to be delegated law enforcement and extensions of the judiciary, using the guise of age verification for child safety but for reasons which have nothing to do with it.
Iain Corby: Yeah, just briefly to add, I think there is a distinction here between when we just accept the parent’s word for the child’s age and when services need to get an independent verification of that age. We do know, this was mentioned earlier, that often, parents are complicit in helping their kids to access services which are age-limited when they shouldn’t be accessing those services. So, sometimes you will need to do an independent age verification rather than simply relying on a parental attestation. So, it’s sort of one step up from self-declaration, but it’s not an independent view of the age of that user.
So Australia is just confirming the point privacy folks have been screaming about for years: age verification is inherently an attack on privacy and security. It will absolutely be used to remove anonymity, decrease security, enhance law enforcement surveillance, and, as the last quote shows, diminish even parental decision-making regarding our children.
Age verification was never going to stop at the age gate. VPNs are just the next thing on the list. Other user empowerment tools (Tor? encrypted DNS?) will be next. There’s simply no version of this that ends with your privacy intact.
