Content Moderation At Scale Is Impossible: Recent Examples Of Misunderstanding Context

from the masnick's-law dept

I've said over and over and over again that content moderation at scale is impossible to do well, and one of the biggest reasons for this is that it's difficult to understand context. Indeed, I've heard that some content moderation policies explicitly suggest that moderators don't try to take context into account. This is not because companies don't think it's important, but the recognition that understanding the context behind every bit of content, would make the process so slow as to be absolutely useless. Yes, it would be great if every content moderator had the time and resources to understand the context of every tweet or Facebook post, but the reality is that we'd then need to employ basically every human being alive to be researching context. Low level content moderators tend to only have a few seconds to make decisions on content, or the entire process slows to a crawl, and then the media will slam those companies for leaving "dangerous" content up too long. So tradeoffs are made, and often that means that understanding context is a casualty of the process. And, of course, that leads to some ridiculous (and hilarious) results. Here are three recent ones that came across my radar. First, someone marketing a tabletop roleplaying game discovered that when they tried to purchase ads on Facebook and Instagram marketing their TTRPG, they had their entire account shut down because they used the word "supplement."

The confusion here is not hard to figure out. First off, lots of roleplaying games have "supplements," or variations/adjustments/add-ons. However... "supplements" also refer to dietary supplements, a market filled with often highly questionable things that people put into their bodies with promises of... well... all sorts of things. And, making matters even worse (as I just discovered!) there's actually a dietary supplement called "RPG" so the Google searches are, well... complex.

And, to make matters even more complex, you may recall that a decade ago, the Justice Department got Google to hand over $500 million for displaying ads for non-approved drugs and supplements. So, I'm sure that both Facebook and Google are extra sensitive to any advertisement that might contain sketchy drugs and supplements -- and thus the rules are designed to be overly aggressive. To them, it's worst case, you shut down an account advertising a role playing game... which is better than having the DOJ show up and confiscate $500 million.

That's not to say this is a good result -- but to explain what likely happened on the back end.

Next up, we have Kate Bevan, who wrote about another content moderation fail on Facebook:

Again, the context here seems fairly obvious. Commenting about a picture of a cat and saying "beautiful puss" is... um... referring to a cat. Not anything else. But, again, in these days when companies are getting sued for all kinds of "bad" things online, you can see why a content moderator having to make a decision in seconds might get this one wrong.

Finally, we've got one that hits a little closer to home. Many of you may be familiar with one of our prolific commenters, That Anonymous Coward (or TAC, for short) who also is a prolific Twitter user. Or was until about a week ago when his account got suspended. Why did his account get suspended? Because of a reply he made to me! Chris Messina had tweeted following the news that Facebook had blocked news links in Australia, by noting that angry Australians were giving bad reviews of Facebook's app in the Australian Apple iOS App Store. And I tweeted, wondering if anyone actually looks at the reviews for apps like Facebook:

If you look below that tweet there are a few replies, including this:

What kind of reply could have possibly violated the rules? Well, here is the offending tweet from TAC:

Reading that in context, it's not at all difficult to see that TAC is mocking people who believe all of those nonsense conspiracy theories. But, right now, Twitter is extra sensitive to conspiracy theories on the site, in part because reporters are highlighting each and every "Q" believer who is allowed to spout nonsense as if it's a moral failing on the part of the companies themselves. So it's perhaps not surprising, even if ridiculous in context, for Twitter to say a tweet like that must violate its rules, and demand that TAC remove it, claiming it violated rules against spreading "misleading and potentially harmful information related to COVID-19."

In this case, TAC appealed... and was (surprisingly quickly) told that his case was reviewed... and the appeal was rejected.

That feels a bit ridiculous, but again highlights the impossibility of content moderation at scale. Technically TAC's tweet is repeating the kinds of disinformation that social media websites are getting attacked over. Of course, it should seem fairly obvious to anyone reading the tweet that he's mocking the people who make those false conspiracy theory claims. But, how do you write a policy that says "unless they're referring to it sarcastically"? Because once you have that in place, and then you get to a point where you have terrible, terrible people saying terrible, terrible things, and then when called on it, they claim they were just saying it "sarcastically."

Indeed, when the "style guide" for the Nazi propaganda site "The Daily Stormer" was leaked, it explicitly told writers to write horrific things with plausible deniability: "it should come across as half-joking." And later in the same document: "The unindoctrinated should not be able to tell if we are joking or not."

That's not to excuse the decisions made here, but to explain how we get to this kind of absurd result. Obviously, it seems to me that all three of these cases are "mistakes" in content moderation, but they're the kind of mistakes that get made when you have to do moderation on millions of pieces of content per day, in a short period of time, or else governments around the world threaten to impose draconian rules or massive fines on you.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: content moderation, content moderation at scale, masnick's impossibility law
Companies: facebook, twitter


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. icon
    That Anonymous Coward (profile), 26 Feb 2021 @ 2:10pm

    Twitter, punishing poor folk they just don't want to deal with.

    I mean I played it off as being a nym means no giving them a cell number, but the truth is I'm fscking poor.

    My life hit massive changes & well cell phone isn't something that I require for quality of life. For less per month then I was paying for the cell I get actual unlimited internet for a single price for a full year.

    Yes I know that believing me comes at a cost, I surround myself in mystery & hyperbole. But Mike & others have seen how losing this social outlet takes a serious toll on me, & being confronted with the idiotic monolith that says oh if you can't provide a number let us know... and then 16 days & multiple reminders I can't provide a number they magically discovered their bot hunting AI tagged me by mistake.

    I'm sure it would have taken longer or never happened but I know people who know people who went to bat for me & suddenly the monolith moved. 16 days & I needed Mike's help (who Jack pays attention to on teh twitters) & Cory Doctrow (yep that famous author guy, I know him too) to get anything to happen.

    How the fsck is the average user supposed to handle this?
    They obviously know that not everyone has a cell phone, yet in practice the system just ignores people pointing out they need to use another path you claim is there but refuse to reveal.

    For the life of me I do not understand Twitters obsession with 'before we can let you serve your punishment, give us your cell phone number so we can text you and prove it is your number.'
    I guess maybe they just want to use those numbers to buy more data about you form the data brokers to see if you are worthy of an account.

    Their intention was to hit me on the nose with a newspaper, you spread the covid misinfo, NAUGHTY, BAP, Now let us text you & THEN we'll let you be chided again & generously let you delete the tweet so that the countdown timer for your sin can begin.
    No interactions with the good people of twitter, only your own tribe for 12 hours... then we'll all sing kumbaya but if you do this again... we might ban you for life!!

    Lets see I've been suspended 3 times now...
    The first 2 they admitted they fscked up & said sorry...
    Crumple up a piece of paper, now smooth it out, tell it sorry... is it pristine again?
    Now I am sitting being punished longer than the MRA asshole who talks about doing horrible things to actual people, he gives them a cell number & is back in 12/24 hours to his cheering fans for having escaped Twitmo.

    I might be an extreme edge case, but am I?
    The monolith will not admit to being a complete shitshow in how their rules are applied & they definitely do not want to admit what their false positive rate is.

    The monolith will not tell us the average wait time for users that fall outside of the get a text & serve your time (they do have a trendy oh we're really busy b/c of the covid so it might take a few extra days... its been a fscking week assholes).

    The monolith will not tell us if there is a review of cases where the gay guy called himself faggot, was banned, appeal denied, appeal denied, & then magically oopsie we messed up.

    I mean I can identify several problems with what happened from the details I know (monolith provided nothing)
    The tweet was 2 yrs old - this should have been a red flag.
    The tweet was reported by multiple mutuals - this should have been a red flag.
    User directly asked for an explanation of how calling himself a bad word was promoting hatred towards others. - The word faggot overruled logic (pay no attention to the results of a search for the word faggot on the platform).

    The monolith will not tell us what happens to bad faith reporters.
    The people who mass reported me, in an attempt to get someone else booted off the platform, we're so excited to take me out... until they figured out I wasn't a sock for their actual target, I was collateral damage in their pissing contest.
    Did they serve a single minute in Twito?
    Why is it okay to lock me out for weeks but people who abused the system got to run around all excited that they took me down?

    I mean I get the fear of revealing any superduper secrets might let bad actors game the system...
    stares at a system being gamed daily anyways

    I joke with Mike & Raccoon Dad (I MISS YOU TIM!!) about moderation here, I manage to trigger it more than the average real person (i think, maybe? Mike have we ever counted?) I stopped being upset by it early on, because the tradeoff of me hitting moderation is the thousands of posts that are spam & stupid that get caught. I've accepted that how I post & interact with the site sometimes smacks me on the nose with a newspaper.
    The biggest difference is if I reach out to TD support, they do something in a reasonable amount of time. (Usually, couple times they've been slow but they have other stuff to take care of they aren't paid just to handle appeals like some monolith employees who sniff glue).
    Hell you can see 2 of my posts this week where I just include the Thanks for kicking this out of moderation line b/c I know I will have triggered it, I'm to fscking old to change how I interact with the site so I accept it. I can see the benefit... I can't see the benefit on the monolith b/c homeopaths put on lab coats and post covid scare information and they are still posting to this day... but I make a snarky comment mocking the insane beliefs & BAM!

    Oh hell what if I found the 1 Trump supporter & offended them by questioning if he actually was the messiah & the covid was just a handy coverup!
    Makes as much sense as any of their decisions in my cases, how many in TwitMo don't have the soapbox I have access to?

    The monolith will never tell... its busy trying to recreate OnlyFans with a 280 character limit...

    Hey Jack, if you see this...
    Know when I get back, the dick pics are gonna start up again.
    The non-penis dick pic protest is the only thing I can do, b/c well your team managing the system are doing a shitty job. If you don't think they are, review my 3 bans & explain how they make ANY sense at all.

    Your team suspended my freind responding to a picture of food for saying it made her want to slap someone.
    She is now suspended for promoting violence towards others.
    By threatening a hypothetical someone.

    Your team smacked another freind with a timeout for her response to a video of someone doing a fake kidnapping for a bday party or something that if someone grabbed her like that she would hit him.
    Wishing ill on others is now a crime on Twitter kids.

    When I get back (I intend to be back, I need the social interaction) expect a Big Brotheresque rebrand of my profile page. If Twitter wants to punish thought-crimes while letting Congresspeople still lie to us with no response... DOUBLEPLUS GOOD! Big Brother is your friend.


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Make this the First Word or Last Word. No thanks. (get credits or sign in to see balance)    
  • Remember name/email/url (set a cookie)

Follow Techdirt
Insider Shop - Show Your Support!

Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.