Again, Algorithms Suck At Determining 'Bad' Content, Often To Hilarious Degrees

from the peachy dept

A few weeks back, Mike wrote a post detailing how absolutely shitty algorithms can be at determining what is "bad" or "offensive" or otherwise "undesirable" content. While his post detailed failings in algorithms judging such weighty content as war-crime investigations versus terrorist propaganda, and Nazi hate-speech versus legitimate news reporting, the central thesis in all of this is that relying on platforms to host our speech and content when those platforms employ very, very imperfect algorithms as gatekeepers is a terrible idea. And it leads to undesirable outcomes at levels far below those of Nazis and terrorism.

Take Supper Mario Broth, for instance. SMB is a site dedicated to fun and interesting information about Nintendo and its history. It's a place that fans go to learn more weird and wonderful information about the gaming company they love. The site also has a Twitter account, which was recently flagged for posting the following tweet.

For the sin of tweeting that image out, the site's entire account was flagged as "sensitive", which means anyone visiting the account was greeted with a warning about how filthy it is. What Twitter's systems thought was offensive about the image, which comes from another video from a costume company that works with Nintendo, is literally anyone's guess. Nobody seems to be able to figure it out. My working theory is that the Princess Peach's lips resemble too closely a more private part of the female anatomy and, when coupled with the flesh-colored face surrounding it sent Twitter's algorithm screaming "Aaah! Vagina!" leading to the flagging of the account. But this is just a guess, because although the "sensitive" flag was eventually removed, SMB never got any response or explanation from Twitter at all.

SMB went as far as to test through dummy accounts whether the image was the entire problem. It was. After posting the image several times from other accounts, each account was flagged within minutes of the posting. It's an algorithm doing this, in other words, and one which seems ill-suited to its task.

What we have here is two related problems. We have a company designed to let speakers speak employing an algorithm to flag offensive content, which it is doing very, very badly. We also have a company with a staff insufficiently capable to correct the errors of its incapable algorithm. This would be annoying in any context other than current reality, which sees rising calls for internet sites to automagically block "bad" content and do so with literally inhuman speed.

That means algorithms. But the algorithms can't do the job. And with sites erring on the side of over-blocking to avoid scrutiny from both the public and governments, that means open communication is the loser in all of this. It's hard to imagine an outcome more anathema to services like Twitter than that.


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    Ninja (profile), 10 Apr 2018 @ 11:17am

    Of course one has to question why humanity has such a problem with penises and vaginas.

    In any case I think we should stop putting content removal in the hands of specific people or algorithms. I'd like to see a decentralized effort where if people enough flagged a determined content it would issue a warning that it was severely flagged but still be visible and after a very high threshold it would be enabled for review by actual humans and possibly removed. Not a perfect system but at least better than what we have now. One can dream, no?

    reply to this | link to this | view in chronology ]

    • icon
      Roger Strong (profile), 10 Apr 2018 @ 12:25pm

      Re:

      Of course one has to question why humanity has such a problem with penises and vaginas.

      Sanity self-preservation. If everything about the Stormy Daniels affair and other Donald Trump sexual misconduct allegations were allowed on TV news - from detailed descriptions to infographics to computer animation - this Presidency would resemble Call of Cthulhu.

      reply to this | link to this | view in chronology ]

      • icon
        Uriel-238 (profile), 10 Apr 2018 @ 12:45pm

        Penises and Vulvas

        Have you seen the new Last Week Tonight bit on CPCs? We really have genitalia issues. Big swooping raging engorged issues.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 10 Apr 2018 @ 1:24pm

          Re: Penises and Vulvas

          It's called stigmatization.

          THE goto tactic for all groups to attack another group is to take a smaller not really a big problem and then stigmatize it until its a huge throbbing.... er problem!

          This was all brought to you by the Catholic Church over the idea that naked people go to hell and clothed people are saved.

          As a Christian, I fully recognize that Christian "institutions" have caused more damage to morals and values than secular ones though they appear to be catching up so they are not outdone.

          Once a group of humans institutionalize something regardless of what that something is it becomes are desired point of infection by evil people to work towards their own ends.

          reply to this | link to this | view in chronology ]

      • icon
        Ninja (profile), 10 Apr 2018 @ 1:20pm

        Re: Re:

        My Pr0n Parody Sense is tingling!

        reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 10 Apr 2018 @ 12:49pm

      Re:

      So, the system in use here?

      reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 11 Apr 2018 @ 12:03am

      Re:

      "Of course one has to question why humanity has such a problem with penises and vaginas."

      *Humanity* doesn't. Western culture seems to have some issues, North American culture more than some others, but less than some others.

      As for the rest of your comment, the problem is that there's a push to make these companies directly and even criminally liable for content that manages to appear on their site even after moderation. A system that would allow questionable material to appear even while moderated would probably fall foul of the laws those people are trying to push through. Similarly, this is why these systems are erring on the side of caution and flagging material is obviously not objectionable - their hand is being forced into making these thing over-zealous rather than let something through.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 10 Apr 2018 @ 12:14pm

    What was the tweet?

    The only pictures I see on the Kotaku page are a highly-pixelated image of Princess Peach (like, 50x30 being shown at 10x that size), and a photo of the author. The Twitter photo link says "Sorry, that page doesn’t exist!". Was the original photo really that pixelated? Or was that just done when reposting, to bypass the filter?

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 10 Apr 2018 @ 12:52pm

      Re: What was the tweet?

      Yeah, I can't find the image either. Really hope someone saved it. I'm curious to see what it was.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 10 Apr 2018 @ 1:17pm

        Re: Re: What was the tweet?

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 10 Apr 2018 @ 2:08pm

          Re: Re: Re: What was the tweet?

          Nothing that loads. Just empty gray boxes.

          reply to this | link to this | view in chronology ]

          • icon
            That One Guy (profile), 11 Apr 2018 @ 9:23pm

            Re: Re: Re: Re: What was the tweet?

            Strange, I just checked the page again and it loaded just fine for me. Might be something on your end.

            reply to this | link to this | view in chronology ]

            • icon
              The Wanderer (profile), 12 Apr 2018 @ 5:35am

              Re: Re: Re: Re: Re: What was the tweet?

              I'm reasonably sure that the people who are reporting problems with such images not loading (properly) are using NoScript or similar, are being passive-aggressive about the fact that Websites which don't have an inherent need to be dynamic should work properly (at a basic level) without JavaScript, and are attempting to hint that giving sites which don't respect that principle a benefit by linking to them is something which should perhaps be avoided.

              I use NoScript myself, and wouldn't allow scripting from such a site just for the purpose of loading images (although I might manually edit page source in Firebug, or potentially do it less-manually with Greasemonkey, to enable myself to view a particular image) - but I haven't followed the links in question, and would have been unlikely to speak up here about the matter anyway. (Although knowing that the linked-to site does have that problem is something that's useful to me.)

              reply to this | link to this | view in chronology ]

  • icon
    Mason Wheeler (profile), 10 Apr 2018 @ 4:04pm

    By Twits, For Twits

    So once again, the Twits at Twitter screw things up for the Twits using Twitter, while the rest of us just sit back and laugh. I cannot for the life of me understand how such an insipid service ever gained even a fraction of the popularity it's achieved...

    reply to this | link to this | view in chronology ]

  • identicon
    Vic, 11 Apr 2018 @ 8:30am

    Nope, no images at any links...

    reply to this | link to this | view in chronology ]

  • identicon
    Just Passin' Thru, 11 Apr 2018 @ 9:01am

    Algorithm problem???

    So, Twitter has an automatic algorithm that scans and mistakenly flags inoffensive images. When it is later reviewed by a human and deemed to be non-offensive, why isn't the image added to a database of known, vetted, inoffensive images that the algorithm checks should it ever encounter that image again?

    reply to this | link to this | view in chronology ]

  • icon
    AntiFish03 (profile), 11 Apr 2018 @ 10:23am

    Honestly I have reported more than a few posts that rant about hang ing all gun owners. Or that gun owners need to be killed and have their firearms taken by force. etc... Facebook refuses to see that as hate speech but it clearly should be. Its also potentially inciting violence against a group of people whose majority is law abiding.
    I have also reported posts that are even more inflammatory and discriminating against a lot of different groups especially the NRA but others as well. FB has given me responses each time that its not hate speech and that if I don't want to see it then I should just hide it. Yet if I were to post a rant about killing anti-gun people it would be pulled nearly instantly for inciting violence etc. Their algorithm is heavily skewed. I think this more than anything is what Cruz was trying to get at. That the bias of what is allowed leans heavily to liberal view points but is exceedingly heavy handed on conservative viewpoints.

    reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 12 Apr 2018 @ 3:54am

      Re:

      Hate speech is generally defined as being against a protected class. Gun fetishist is not a protected class. It's no more hate speech than people saying that all Pokemon fans should be shot. It might be offensive, even something that authorities should be aware of, but not hate speech according to the usual guidelines.

      "That the bias of what is allowed leans heavily to liberal view points but is exceedingly heavy handed on conservative viewpoints."

      Which conservative viewpoints? Honestly held beliefs about honest conservative values, or the racist, misogynistic, homophobic, etc., viewpoints often espoused by those who claim to be conservative? There's a big difference, and if someone is blocking those viewpoints it's not anti-conservative bias if all the racist, homophobic woman haters happen to be on that side.

      Give me examples of non-hatred being blocked and I might agree. But, if the majority of hatred is on your "team", you may wish to examine that "team" rather than complaining about censorship. If 90% of objectionable material is in the hands of one "team", it's not unfair if they get 90% of the blocks.

      reply to this | link to this | view in chronology ]

  • icon
    Uriel-238 (profile), 12 Apr 2018 @ 10:52am

    "Protected Classes"

    It's entirely off topic, though it is a problem that we have:

    Large amounts of accepted hate speech is against classes who aren't yet (or aren't quite) protected, whether countercultures, disabled, obese people, or the biggest one, poor people, even though there are clear trends of discrimination against them. (Trump's new welfare reform serving as a notable recent example.)

    It's a problem that we classify one form of bigotry over another rather than defining specific actions that are universally regarded as acts of hatred. As a result, society's level of awareness is always many steps behind.

    reply to this | link to this | view in chronology ]

  • identicon
    Châu, 17 Apr 2018 @ 6:29pm

    Open Source

    If open source algorithm, easier test and find it problems.

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: Techdirt Logo Gear
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.