Why Making Social Media Companies Liable For User Content Doesn’t Do What Many People Think It Will

Content Moderation

from the how-stuff-works dept

Brazil’s Supreme Court appears close to ruling that social media companies should be liable for content hosted on their platforms—a move that appears to represent a significant departure from the country’s pioneering Marco Civil internet law. While this approach has obvious appeal to people frustrated with platform failures, it’s likely to backfire in ways that make the underlying problems worse, not better.

The core issue is that most people fundamentally misunderstand both how content moderation works and what drives platform incentives. There’s a persistent myth that companies could achieve near-perfect moderation if they just “tried harder” or faced sufficient legal consequences. This ignores the mathematical reality of what happens when you attempt to moderate billions of pieces of content daily, and it misunderstands how liability actually changes corporate behavior.

Part of the confusion, I think, stems from people’s failure to understand the impossibility of doing content moderation well at scale. There is a very wrong assumption that social media platforms could do perfect (or very good) content moderation if they just tried harder or had more incentive to do better. Without denying that some entities (*cough* ExTwitter *cough*) have made it clear they don’t care at all, most others do try to get this right, and discover over and over again how impossible that is.

Yes, we can all point to examples of platform failures that are depressing and seem obvious that things should have been done differently, but the failures are not there because “the laws don’t require it.” The failures are because it’s impossible to do this well at scale. Some people will always disagree with how a decision comes out, and other times there are no “right” answers. Also, sometimes, there’s just too much going on at once, and no legal regime in the world can possibly fix that.

Given all of that, what we really want are better overall incentives for the companies to do better. Some people (again, falsely) seem to think the only incentives are regulatory. But that’s not true. Incentives come in all sorts of shapes and sizes—and much more powerful than regulations are things like the users themselves, along with advertisers and other business partners.

Importantly, content moderation is also a constantly moving and evolving issue. People who are trying to game the system are constantly adjusting. New kinds of problems arise out of nowhere. If you’ve never done content moderation, you have no idea how many “edge cases” there are. Most people—incorrectly—assume that most decisions are easy calls and you may occasionally come across a tougher one.

But there are constant edge cases, unique scenarios, and unclear situations. Because of this, every service provider will make many, many mistakes every day. There’s no way around this. It’s partly the law of large numbers. It’s partly the fact that humans are fallible. It’s partly the fact that decisions need to be made quickly without full information. And a lot of it is that those making the decisions just don’t know what the “right” approach is.

The way to get better is constant adjusting and experimenting. Moderation teams need to be adaptable. They need to be able to respond quickly. And they need the freedom to experiment with new approaches to deal with bad actors trying to abuse the system.

Putting legal liability on the platform makes all of that more difficult

Now, here’s where my concerns about the potential ruling in Brazil get to: if there is legal liability, it creates a scenario that is actually less likely to lead to good outcomes. First, it effectively requires companies to replace moderators with lawyers. If your company is now making decisions that come with significant legal liability, that likely requires a much higher type of expertise. Even worse, it’s creating a job that most people with law degrees are unlikely to want.

Every social media company has at least some lawyers who work with their trust & safety teams to review the really challenging cases, but when legal liability could accrue for every decision, it becomes much, much worse.

More importantly, though, it makes it way more difficult for trust & safety teams to experiment and adapt. Once things include the potential of legal liability, then it becomes much more important for the companies to have some sort of plausible deniability—some way to express to a judge “look, we’re doing the same thing we always have, the same thing every company has always done” to cover themselves in court.

But that means that these trust & safety efforts get hardened into place, and teams are less able to adapt or to experiment with better ways to fight evolving threats. It’s a disaster for companies that want to do the right thing.

The next problem with such a regime is that it creates a real heckler’s veto-type regime. If anyone complains about anything, companies are quick to take it down, because the risk of ruinous liability just isn’t worth it. And we now have decades of evidence showing that increasing liability on platforms leads to massive overblocking of information. I recognize that some people feel this is acceptable collateral damage… right up until it impacts them.

This dynamic should sound familiar to anyone who’s studied internet censorship. It’s exactly how China’s Great Firewall originally operated—not through explicit rules about what was forbidden, but by telling service providers that the punishment would be severe if anything “bad” got through. The government created deliberate uncertainty about where the line was, knowing that companies would respond with massive overblocking to avoid potentially ruinous consequences. The result was far more comprehensive censorship than direct government mandates could have achieved.

Brazil’s proposed approach follows this same playbook, just with a different enforcement mechanism. Rather than government officials making vague threats, it would be civil liability creating the same incentive structure: when in doubt, take it down, because the cost of being wrong is too high.

People may be okay with that, but I would think that in a country with a history of dictatorships and censorship, they would like to be a bit more cautious before handing the government a similarly powerful tool of suppression.

It’s especially disappointing in Brazil, which a decade ago put together the Marco Civil, an internet civil rights law that was designed to protect user rights and civil liberties—including around intermediary liability. The Marco Civil remains an example of more thoughtful internet lawmaking (way better than we’ve seen almost anywhere else, including the US). So this latest move feels like backsliding.

Either way, the longer-term fear is that this would actually limit the ability of smaller, more competitive social media players to operate in Brazil, as it will be way too risky. The biggest players (Meta) aren’t likely to leave, but they have buildings full of lawyers who can fight these lawsuits (and often, likely, win). A study we conducted a few years back detailed how as countries ratcheted up their intermediary liability, the end result was, repeatedly, fewer online places to speak.

That doesn’t actually improve the social media experience at all. It just gives more of it to the biggest players with the worst track records. Sure, a few lawsuits may extract some cash from these companies for failing to be perfect, but it’s not like they can wave a magic wand and not let any “criminal” content exist. That’s not how any of this works.

Some responses to issues raised by critics

When I wrote about this on a brief Bluesky thread, I received hundreds of responses—many quite angry—that revealed some common misunderstandings about my position. I’ll take the blame for not expressing myself as clearly as I should have and I’m hoping the points above lay out the argument more clearly regarding how this could backfire in dangerous ways. But, since some of the points were repeated at me over and over again (sometimes with clever insults), I thought it would be good to address some of the arguments directly:

But social media is bad, so if this gets rid of all of it, that’s good. I get that many people hate social media (though, there was some irony in people sending those messages to me on social media). But, really what most people hate is what they see on social media. And as I keep explaining, the way we fix that is with more experimentation and more user agency—not handing everything over to Mark Zuckerberg and Elon Musk or the government.

Brazil doesn’t have a First Amendment, so shut up and stop with your colonialist attitude. I got this one repeatedly and it’s… weird? I never suggested Brazil had a First Amendment, nor that it should implement the equivalent. I simply pointed out the inevitable impact of increasing intermediary liability on speech. You can decide (as per the comment above) that you’re fine with this, but it has nothing to do with my feelings about the First Amendment. I wasn’t suggesting Brazil import American free speech laws either. I was simply pointing out what the consequences of this one change to the law might create.

Existing social media is REALLY BAD, so we need to do this. This is the classic “something must be done, this is something, we will do this” response. I’m not saying nothing must be done. I’m just saying this particular approach will have significant consequences that it would help people to think through.

It only applies to content after it’s been adjudicated as criminal. I got that one a few times from people. But, from my reading, that’s not true at all. That’s what the existing law was. These rulings would expand it greatly from what I can tell. Indeed, the article notes how this would change things from existing law:

The current legislation states social media companies can only be held responsible if they do not remove hazardous content after a court order.

[….]

Platforms need to be pro-active in regulating content, said Alvaro Palma de Jorge, a law professor at the Rio-based Getulio Vargas Foundation, a think tank and university.

“They need to adopt certain precautions that are not compatible with simply waiting for a judge to eventually issue a decision ordering the removal of that content,” Palma de Jorge said.

You’re an anarchocapitalist who believes that there should be no laws at all, so fuck off. This one actually got sent to me a bunch of times in various forms. I even got added to a block list of anarchocapitalists. Really not sure how to respond to that one other than saying “um, no, just look at anything I’ve written for the past two and a half decades.”

America is a fucking mess right now, so clearly what you are pushing for doesn’t work. This one was the weirdest of all. Some people sending variations on this pointed to multiple horrific examples of US officials trampling on Americans’ free speech, saying “see? this is what you support!” as if I support those things, rather than consistently fighting back against them. Part of the reason I’m suggesting this kind of liability can be problematic is because I want to stop other countries from heading down a path that gives governments the power to stifle speech like the US is doing now.

I get that many people are—reasonably!—frustrated about the terrible state of the world right now. And many people are equally frustrated by the state of internet discourse. I am too. But that doesn’t mean any solution will help. Many will make things much worse. And the solution Brazil is moving towards seems quite likely to make the situation worse there.

Filed Under: brazil, content moderation, free speech, impossibility, intermediary liability, marco civil, platform liability, social media