Fan Uses AI Software To Lipread What Actors Really Said In TV Series Before Chinese Authorities Censored Them

from the I-saw-what-you-said-there dept

It's hardly news to Techdirt readers that China carries out censorship on a massive scale. What may be more surprising is that its censorship extends to even the most innocuous aspects of life. The ChinAI Newsletter, which provides translations by Jeff Ding of interesting texts from the world of Chinese AI, flags up one such case. It concerns a Chinese online TV series called "The Bad Kids". Here's how the site Sixth Tone describes it:

Since its first episodes were released on China's Netflix-like video platform iQiyi in mid-June, "The Bad Kids" has earned sweeping praise for its plot, cinematography, casting, dialogue, pacing, and soundtrack. It's also generated wide-ranging online discussion on human nature due to the psychology and complex motivations of its characters.

However, as the Sixth Tone article points out, the authorities required "a lot of changes" for the series to be approved. One fan of "The Bad Kids", Eury Chen, wanted to find out what exactly had been changed, and why that might be. In a blog post translated by ChinAI, Chen explained how he went about this:

Two days ago, I watched the TV series "The Bad Kids" in one go, and the plot was quite exciting. The disadvantage is that in order for the series to pass the review (of the National Radio and Television Administration), the edited sequences for episodes 11 and 12 were disrupted, even to the point that lines were modified, so that there are several places in the film where the actor's mouth movements and lines are not matched, which makes the plot confusing to people. Therefore, I tried to restore the modified lines through artificial intelligence technology, thereby restoring some of the original plot, which contained a darker truth.

The AI technology involved using Google's Facemesh package, which can track key "landmarks" on faces in images and videos. By analyzing the lip movements, it is possible to predict the sounds of a Chinese syllable. However, there is a particular problem that makes it hard to lipread Chinese using AI. There are many homophones in Chinese (similar sounds, different meanings). In order to get around this problem, Chen explored the possible sequences of Chinese characters to find the ones that best match the plot at that point. As his blog post (and the ChinAI translation) explains, this allowed him to work out why certain lines were blocked by the Chinese authorities -- turns out it was for totally petty reasons.

Perhaps more interesting than the details of this particular case, is the fact that it was possible to use AI to carry out most of the lipreading, leaving human knowledge to choose among the list of possible Chinese phrases. Most languages don't require that extra stage, since they rarely have the same number of homophones that Chinese does. Indeed, for English phrases, researchers already claimed in 2016 that their AI-based LipNet achieved "95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders".

It's clear that we are fast approaching a situation where AI is able to lipread a video in any language. That is obviously a boon for the deaf or hard of hearing, but there's a serious downside. It means that soon all those millions of high-quality CCTV systems around the world will not only be able to use facial recognition software to work out who we are, but also run AI modules to lipread what we are saying.

Follow me @glynmoody on Twitter, Diaspora, or Mastodon.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: ai, censorship, china, facial recognition, lip reading


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    Boba Fat (profile), 15 Jul 2020 @ 7:40pm

    AI lipreading

    Masks: They're not just for Covid-19 anymore!

    reply to this | link to this | view in chronology ]

  • icon
    That One Guy (profile), 15 Jul 2020 @ 9:27pm

    I foresee a massive growth(in both uses of the word) in mustaches in the near future. Large, bushy, lip-covering mustaches...

    reply to this | link to this | view in chronology ]

    • icon
      Scary Devil Monastery (profile), 16 Jul 2020 @ 1:57am

      Re:

      ...and a penchant for actors to speak while turned away from the camera. A renaissance of the mouth-covering veil for the ladies and the mysterious mouth-covering scarf a la "The Shadow" for the gents.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 16 Jul 2020 @ 9:34am

        Re: Re:

        Why bother? If they dubbed out the prohibited lines as poorly as described, the logical next step is that next time they poorly dub out the lines and apply an equally crude visual censor to the actor's mouth - the classic black bar used for nudity, or the hyper-pixelization used to obscure a face.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 16 Jul 2020 @ 5:56pm

          Re: Re: Re:

          In some places they will probably end up dubbing everyone involved out of existence

          reply to this | link to this | view in chronology ]

        • icon
          Scary Devil Monastery (profile), 17 Jul 2020 @ 7:09am

          Re: Re: Re:

          "Why bother? If they dubbed out the prohibited lines as poorly as described..."

          With the soviets you could sort of predict that the propaganda and censorship was heavyhanded to the point of absurdity. With China it's a bit of hit and miss - some of their PR stunts growing on the midden of the blunt and obvious old school copypasta of political manifestos - and some of their more subtle PR is good enough gaslighting to convince even the more sceptical for a while.

          reply to this | link to this | view in chronology ]

  • identicon
    Bruce C., 15 Jul 2020 @ 9:33pm

    CCTV lip-reading.

    Then it's a good thing we're all already wearing masks. Right? Right? (crickets).

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 15 Jul 2020 @ 10:44pm

    Time to start speaking like a bad kung fu dub.

    reply to this | link to this | view in chronology ]

  • identicon
    Daydream, 16 Jul 2020 @ 1:51am

    Excuse me, this article is about how lip-reading software was used to subvert a third-party's petty attempt at censorship.
    Why are the comments so far about trying to prevent lip-reading from working?

    reply to this | link to this | view in chronology ]

    • icon
      Scary Devil Monastery (profile), 16 Jul 2020 @ 1:59am

      Re:

      "Why are the comments so far about trying to prevent lip-reading from working?"

      More like the generally highly cynical crowd of Techdirt commenters are predicting the response of chinese censors to lip-reading.

      reply to this | link to this | view in chronology ]

      • icon
        Uriel-238 (profile), 16 Jul 2020 @ 2:04am

        Also because we're scared of surveillance

        Here in the states, we've become acutely aware of how often law enforcement is trying to listen in on communication, and lip-reading AI will add to their repertoire of techniques.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 16 Jul 2020 @ 2:12am

          Re: Also because we're scared of surveillance

          And give them another tool for generating probable cause after they have selected a target.

          reply to this | link to this | view in chronology ]

          • icon
            Scary Devil Monastery (profile), 20 Jul 2020 @ 5:03am

            Re: Re: Also because we're scared of surveillance

            "...another tool for generating probable cause..."

            Because their current "He's breathing oxygen" hasn't served them well until now?

            reply to this | link to this | view in chronology ]

  • icon
    Uriel-238 (profile), 16 Jul 2020 @ 2:08am

    Curiously petty censorship is a problem here in the states

    Rare government documents that are censored for declassification twice have revealed different reviewers will black out different parts of the document. Many of our departments lack consistent standards for declassification (and it follows, for classification).

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 16 Jul 2020 @ 2:59pm

      Re: Curiously petty censorship is a problem here in the states

      Many of our departments lack consistent standards for declassification (and it follows, for classification).

      Perhaps. It's also quite possible they do have consistent standards, but like pretty much all other sets of standards for moderation, different people with access to the same training and information will still interpret the same set of standards in wildly different and yet still reasonable ways.

      reply to this | link to this | view in chronology ]

      • icon
        Uriel-238 (profile), 16 Jul 2020 @ 3:08pm

        Re: Re: Curiously petty censorship is a problem here in the stat

        If different people with access to the same training and information are interpreting the same set of standards in wildly different and yet still reasonable ways, and is inconsistent in its application, then the standard fails to be consistent.

        SEE ALSO: The holy bible.

        reply to this | link to this | view in chronology ]

      • icon
        David (profile), 19 Jul 2020 @ 5:03pm

        Re: Re: Curiously petty censorship is a problem here in the stat

        Then it isn't a standard. If it is standard, it will always have the same results.

        reply to this | link to this | view in chronology ]

  • This comment has been flagged by the community. Click here to show it
    icon
    BT Mail (profile), 16 Jul 2020 @ 4:02am

    BT Internet Email

    Hi , Just wanted to say thanks for this fantastic article.
    Website: http://btmaills.com

    reply to this | link to this | view in chronology ]

  • identicon
    Jim, 16 Jul 2020 @ 4:09am

    Err?

    In reply, another interesting factoid, with the lipreading ai, there comes a face reading ai, and a deep fake ai. Now they do not have to catch you, they just have to modify your patterns to a blank screen, now which is real?

    reply to this | link to this | view in chronology ]

  • identicon
    Pivelation, 16 Jul 2020 @ 4:47am

    Read my lips

    No new taxes!

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 16 Jul 2020 @ 6:51am

    We've had lipreading computers since 2001.

    reply to this | link to this | view in chronology ]

  • identicon
    Rekrul, 16 Jul 2020 @ 11:36am

    Most languages don't require that extra stage, since they rarely have the same number of homophones that Chinese does. Indeed, for English phrases, researchers already claimed in 2016 that their AI-based LipNet achieved "95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders".

    So the AI can do what 90% of the people posting on the internet can't?

    Because it seems that absolutely nobody today knows the difference between "your" and "You're", or "its" and "it's". Hell, they don't even know the difference between "there", "their" and "they're" most of the time, or that "of" isn't a replacement for "have" when it follows a word ending in "ould".

    reply to this | link to this | view in chronology ]

    • icon
      That One Guy (profile), 16 Jul 2020 @ 1:19pm

      Re:

      Oh you would love Kingdom of Loathing's 'literacy test'...

      reply to this | link to this | view in chronology ]

    • icon
      Uriel-238 (profile), 16 Jul 2020 @ 3:12pm

      Common homophones

      Some advanced word-processing and copy editing software will scan for homophones and highlight the ones whose meanings seem to be inconsistent with context.

      So yes, this is a thing AI can do better than humans, but still a far reach from perfectly.

      Maybe as we teach kids how to talk to their personal digital assistant software (e.g. SIRI, Cortana, Google Assistant) the language will develop towards something computers can more easily understand.

      reply to this | link to this | view in chronology ]

      • identicon
        Rekrul, 22 Jul 2020 @ 10:14am

        Re: Common homophones

        Some advanced word-processing and copy editing software will scan for homophones and highlight the ones whose meanings seem to be inconsistent with context.

        In the above sentence, probably 90% of today's users would have written "who's".

        reply to this | link to this | view in chronology ]

        • icon
          Uriel-238 (profile), 22 Jul 2020 @ 12:45pm

          "90% of today's users would have written 'who's'"

          And I would have cringed reading it. But I too sometimes get confused by the pronoun possessive rules, usually when my coffee intake has been insufficient.

          reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 16 Jul 2020 @ 6:00pm

      Re:

      It's even more fun when we get to common phrases. Words get substituted, showing that people (mis)using such phrases do not even understand the meaning of words, they merely parrot things.

      reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Make this the First Word or Last Word. No thanks. (get credits or sign in to see balance)    
  • Remember name/email/url (set a cookie)

Close

Add A Reply

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Make this the First Word or Last Word. No thanks. (get credits or sign in to see balance)    
  • Remember name/email/url (set a cookie)

Follow Techdirt
Special Affiliate Offer

Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.