Techdirt's think tank, the Copia Institute, is working with the Trust & Safety Professional Association and its sister organization, the Trust & Safety Foundation, to produce an ongoing series of case studies about content moderation decisions. These case studies are presented in a neutral fashion, not aiming to criticize or applaud any particular decision, but to highlight the many different challenges that content moderators face and the tradeoffs they result in. Find more case studies here on Techdirt and on the TSF website.

Creating Family Friendly Chat More Difficult Than Imagined (1996)

from the the-kids-will-find-a-way dept

Summary: Creating family friendly environments on the internet presents some interesting challenges that highlight the trade-offs in content moderation. One of the founders of Electric Communities, a pioneer in early online communities, gave a detailed overview of the difficulties in trying to build such a virtual world for Disney that included chat functionality. He described being brought in by Disney alongside someone from a kids’ software company, Knowledge Adventure, who had built an online community in the mid-90s called “KA-Worlds.” Disney wanted to build a virtual community space, HercWorld, to go along with the movie Hercules. After reviewing Disney’s requirements for an online community, they realized chat would be next to impossible:

Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?"

The response was one I will never forget: "Disney’s standard is quite clear:

No kid will be harassed, even if they don’t know they are being harassed."...

"OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied.

One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?"

Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds."

"We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world."

"We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence:

I want to stick my long-necked Giraffe up your fluffy white bunny.

In that initial 1996 project, chat was abandoned, but as they continued to develop HercWorld, they quickly realized that they still had to worry about chat, even without a chat feature:

It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff… Oops, what was that last thing again?

"…kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.", one of the designers said.

I couldn’t resist, "Umm. Doesn’t that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You’ve just invented Block-ChatTM. If you can put down objects, you’ve got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed." We all laughed, but it was that kind of awkward laugh that you know means that we’re all probably just wasting our time.

Decisions for family-friendly community designers:

  • Is there a way to build a chat that will not be abused by clever kids to reference forbidden content (e.g., swearing, innuendo, harassment, abuse)?
  • Can you build a chat that does not require universal moderation and pre-approval of everything that users will say?
  • Are there ways in which kids will still be to communicate with others even without an actual chat feature?
  • How much of a “community” do you have with no chat or extremely limited chat?

Questions and policy implications to consider:

  • Is it possible to create an online family friendly environment that will work?
    • If so how do you prevent abuse?
    • If not, how do you handle the fact that kids will get online whether they are allowed to or not?
  • How do you incentivize companies to create spaces that actually remain as child-friendly as possible?
  • If “the kids will always find a way” to get around limitations, does it make sense to hold the companies themselves responsible?
  • Should family friendly environments require full-time monitoring, or pre-vetting of any usage?
Resolution: Disney eventually abandoned the idea of HercWorld due to all of the issues raised. However, the interview highlights the fact that they tried again a couple of years later, with an online chat where users could only pull from a pre-selected list of sentences, but it did not have much success:

"The Disney Standard" (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.

Brian had an idea though: Fully pre-constructed sentences – dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature "Chatless Chat." So, we built and launched it for them. Disney was still very tentative about the genre, so they only ran it for about six months; I doubt it was ever very popular.

The same interview notes that Disney tried once again in 2002 with a new world called “ToonTown”, with pulldown menus that allowed you to construct very narrowly tailored speech within the chat to try to avoid anything that violated the rules.

As the story goes, Disney still had problems with this. To make sure people were only communicating with people they knew in real life, one of the restrictions in this new world was that you had to have a secret code from any user you wished to chat with. The thinking was that parents would print these out for kids who could then share them with their friends in real life, and they could link up and “chat” in the online world.

And yet, once again, people figured out how to get around the restrictions:

Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:

User A:"Please be my friend."
User A:"Come to my house?"
User B:"Okay."
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:"Okay"
B:[Writes down 4 on a piece of paper and says] "Okay."
A:[Move objects to make the next letter/number in the code] "Okay"
B:[Writes…] "Okay"
A:[Remove objects to represent a "space" in the code] "Okay"
[Repeat steps as needed, until…]
A:"Okay"
B:[Enters secret code into Toontown software.]
B:"There, that worked. Hi! I’m Jim 15/M/CA, what’s your A/S/L?"

Incredibly, there was an entire Wiki page on the Disney Online Worlds domain that included a variety of other descriptions on how to exchange your secret number within the game, even as users were not supposed to be doing so:

For example, let's say you have a secret code (1hh 5rj) which you would like to give to a toon named Bob.

First, you should make clear that you want to become their SF.
You: Please be my friend!
You: (random SF chat)
You: I can't understand you
You: Let's work on that
Bob: Yes
Now, start the secret.
You: (Jump 1 time and say OK. Jump 1 time because that is the first thing in your code. Say OK to confirm that was part of your secret.)
Bob: OK (Wait for this, as this means he has written down or otherwise recorded the 1)
You: Hello! OK (Say hello because the first letter of hello is h, which is the second part of your secret.)
Bob: OK (again, wait for confirmation)
Repeat above step, as you have the same letter for the third part of your secret.
Bob: OK (by now you should know to wait for this)
You: (Jump 5 times and say OK. Jump 5 times as this is the 4th part of your secret)
Bob: OK
You: Run! OK (The 5th part of your secret is r, and "Run!" starts with r)
Bob: OK
You: Jump! OK (Say this because j is the last part of your secret.)
Bob: OK
At this point, you have successfully transmitted the code to Bob.
Most likely, Bob will understand, and within seconds, you will be Secret Friends!

So even though Disney eventually did enable a very limited chat, with strict rules to keep people safe, it still left open many challenges for early trust & safety work.

Images from HabitChronicles

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: chat, content moderation, family friend
Companies: disney


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    Samuel Abram (profile), 12 Aug 2020 @ 1:47pm

    Nintendo's experiments

    Would Nintendo's dabbling into Friend Codes be similar as far as content moderation goes? Because whereas in the past you could send drawings of penises over DS pictochat, in today's era of the Nintendo Switch, there is no internet browser or chat function, but there are still friend codes.

    Could Nintendo's experiments be considered more successful or proof that "family-friendly chat" is as impossible as absolute zero or psychic powers?

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 12 Aug 2020 @ 3:57pm

      Re: Nintendo's experiments

      Could Nintendo's experiments be ... proof that "family-friendly chat" is as impossible as ... psychic powers?

      What? I have VAST psychic powers! I can, with my psychic powers, cause you intense nausea, provoke anger, and even cause you to (try to) make my comments disappear. All I have to do, using my mind alone, is ...

      ... cause my fingers to type: I still believe in Trump.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 12 Aug 2020 @ 5:40pm

        Re: Re: Nintendo's experiments

        er, what?

        reply to this | link to this | view in chronology ]

      • icon
        Scary Devil Monastery (profile), 14 Aug 2020 @ 12:52am

        Re: Re: Nintendo's experiments

        "... cause my fingers to type: I still believe in Trump."

        No, that in itself only causes us to doubt your sanity and credibility as a commenter.

        You might as well have been writing that you still believe in Stalin, Mao or Hitler. We aren't going to be shocked as much as saddened for your sake.

        reply to this | link to this | view in chronology ]

    • identicon
      PatrickH, 13 Aug 2020 @ 1:34am

      Re: Nintendo's experiments

      At least according to a leaked presentation friend codes came about cause making users choose unique names and others typing them in wasn't simple enough. Though I imagine avoiding offensive names is a benefit that if not realized during design, is one that keeps it in place.

      https://www.eurogamer.net/articles/2020-05-04-nintendo-chose-internet-friend-codes-because-us ing-real-names-was-not-simple-enough

      Possibly, only Nintendo knows why they got rid of it. Though a chat so limited by design it's "safe" is probably not worth using, and human moderation is quickly overwhelmed. There might be hope if AI becomes so advanced it rivals humans in understanding human communication and fast enough to effective moderate, but that's a pretty big if, and it seems highly unlikely in the foreseeable future.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 12 Aug 2020 @ 4:04pm

    FUCK FUCK FUCK FUCK FUCK
    FUCKEN GODDAM KIDS RUIN EVERYTHING

    reply to this | link to this | view in chronology ]

  • icon
    Ehud Gavron (profile), 12 Aug 2020 @ 4:49pm

    1hh5r?

    Now I have to go change all my banking passwords. Techdirt/Copia how dare you out my secure password!!

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 12 Aug 2020 @ 5:40pm

    ...people would turn 100 Afro-Heads into a waterbed...

    I'm sorry, what the actual fuck?

    reply to this | link to this | view in chronology ]

  • identicon
    Lewdtoo, 12 Aug 2020 @ 6:43pm

    "How can we prevent people from making lewd/innapropriate comments?"

    "Enough nukes in the centre of the earth core could do the trick"

    some days later

    "No dice, we had our people ran simulations"

    "Where's the problem?"

    "They managed to have the debris spell out 'F U C K you alien scum' and form a penis"

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 12 Aug 2020 @ 7:06pm

    Task is insanity like DRM

    Really the very concept is futile like DRM because it requires a contradiction - to be able to communicate and not communicate at the same time. With arbitrary mapping they can easily send dirty morse code to go to absurd extremes.

    The closest that can be done is to make it more obtuse like the passing of codes. Although it might work for covering their asses if the moderators remain oblivious to sexual RP done by items in trade windows and emotes even if "large doll and eggplant for small doll" followed up by adding blue eyedrops with the doll" means something deeply disturbing.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 12 Aug 2020 @ 9:36pm

    Here's the start of the second paragraph of Shannon's information theory paper:

    The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning

    The meaning is, you'll notice, separate from the message. If you can communicate anything then you can communicate anything because it's only a matter of assigning different meaning to the information you can push through the channel.

    reply to this | link to this | view in chronology ]

  • icon
    PaulT (profile), 12 Aug 2020 @ 11:11pm

    "Is it possible to create an online family friendly environment that will work?
    If so how do you prevent abuse?"

    Yes.
    By having parents actually do their jobs and actively parent their children rather than using passive entertainment as a babysitter.

    "Should family friendly environments require full-time monitoring, or pre-vetting of any usage?"

    Yes.

    reply to this | link to this | view in chronology ]

    • icon
      nasch (profile), 13 Aug 2020 @ 8:39am

      Re:

      By having parents actually do their jobs and actively parent their children rather than using passive entertainment as a babysitter.

      Requiring parents to sit by their children and watch the screen for everything going on the entire time they're using the system does not violate any known laws of physics, so it's not impossible. But it's also not going to happen.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 13 Aug 2020 @ 9:05am

        Re: Re:

        There are web filters than can be installed on home networks, which do at least a reasonable, but not foolproof, job. Concerned parents should look into using them, rather than trying to get companies to ruin the Internet by making it totally child safe.

        reply to this | link to this | view in chronology ]

        • icon
          Scary Devil Monastery (profile), 14 Aug 2020 @ 12:59am

          Re: Re: Re:

          "Concerned parents should look into using them, rather than trying to get companies to ruin the Internet by making it totally child safe."

          Ah, yes, parental filters. Children have great sport in circumventing those.

          I mainly encourage the use of such because it means more children get a head start in understanding computer security.

          reply to this | link to this | view in chronology ]

          • icon
            PaulT (profile), 14 Aug 2020 @ 1:15am

            Re: Re: Re: Re:

            "Children have great sport in circumventing those."

            That's one part of the problem. The other part is the "Scunthorpe problem", where over-filtering will remove access to perfectly innocent content, including things that might actually be necessary for the child's education. So, either they bypass the filter making it useless, or it blocks them from doing their homework - or, perhaps both as a kid willing going in search of porn probably won't mind being told he can't do school work.

            reply to this | link to this | view in chronology ]

            • identicon
              Anonymous Coward, 14 Aug 2020 @ 3:19am

              Re: Re: Re: Re: Re:

              The other part is the "Scunthorpe problem",

              I would rather some parents had to deal with that than the vocal minorities impose it on everybody else.

              reply to this | link to this | view in chronology ]

      • icon
        PaulT (profile), 13 Aug 2020 @ 11:25pm

        Re: Re:

        It will happen long before the web will be safe for unsupervised children.

        reply to this | link to this | view in chronology ]

        • icon
          Scary Devil Monastery (profile), 14 Aug 2020 @ 1:03am

          Re: Re: Re:

          ...as in; "Never", you mean.

          What really gets to me is that the same "concerned" parents dreading the moral hazard of the internet usually don't have much trouble sending their children to congregate in vast herds of...well, other children.

          At some point you'd think it'd be obvious that if you want your kids to be safe what you need to do is tell them of the dangers, how to recognize them, and how to get away from them. Instead too many parents just do their damnedest to keep their kids from learning that the world is full of dangers.

          reply to this | link to this | view in chronology ]

          • icon
            PaulT (profile), 14 Aug 2020 @ 1:18am

            Re: Re: Re: Re:

            "...as in; "Never", you mean."

            Both are "never" situations in the broader scheme of things. But, between "censorship of adults working without fail" and "parents taking an active interest in their child's welfare", at least the latter will occasionally happen in some households.

            "At some point you'd think it'd be obvious that if you want your kids to be safe what you need to do is tell them of the dangers, how to recognize them, and how to get away from them."

            ...but that takes effort! How is the parent who uses YouTube as a babysitter for their 3 year old meant to do that without lifting a finger?!?

            reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 13 Aug 2020 @ 5:12am

    This is basically a given in most games. Even when they try to give baked-in communication phrases, there will always be a way to imply something bad-mannered. Examples: Hearthstone - "Well Played!" spam League of Legends - lots of possible pings here, one of them is a literal question mark that appears on the map/mini-map FPS games - teabagging "The convoluted wording of legalisms grew up around the necessity to hide from ourselves the violence we intend toward each other. Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. You have done violence to him, consumed his energy."

    reply to this | link to this | view in chronology ]

  • identicon
    Pixelation, 13 Aug 2020 @ 6:34am

    "Is there a way to build a chat that will not be abused by clever kids...?"

    Yes. However, they can never be allowed to use it for it to work.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 13 Aug 2020 @ 4:09pm

    I feel Club Penguin managed to do it "right". First a word filter for edgy kids who log in and type "fuck my bitch up" or whatever. After that, manual moderation of messages and a report function. It seems that until we have true AI, there is no alternative to just sitting people in front of the screen and having them moderate messages.

    More philosophically however, what age is appropriate for letting a person figure out that the world is full of shitheads? 21? 15? 12? In real life I could say whatever shit I wanted to whatever other kid I wanted outside of school and believe me we said a LOT of shit. I get that parents want their kids to be safe, but isn't online inherently safe? Give people the tools to block others, have a system to catch ban evaders (you need this anyway), have a filter that doesn't let anything that looks like a credit card number or an address through (you also need this anyway), and I dare say that's enough for a bunch of 12-16 year olds.

    For younger kids, the Disney system seems ok. Yes, you can still transmit information through various means, but whatever you do, any sufficiently determined child will not be deterred. At some point you have to say "we made the best system we could, if you decide to break the rules, you're on your own".

    It's weird this Internet thing. On one hand it sounds scary that you can just connect anyone with anyone at random with no overseers. On the other, you can always refuse to receive messages from anyone and services try to keep your personal info secure so strangers don't have direct access to it. It sounds like just about anyone can just harm your kid because they're online, but in reality it's more like you can hear the yells of the crazy guy from across town if you listen carefully, and you can always just not listen.

    reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 13 Aug 2020 @ 11:25pm

      Re:

      Basically, some people think that the Internet should be Disney World, with everything cuddly and child safe and lots of helpful people to bow to your every wish should you have a problem with any aspect of life.

      In reality, the internet is like any major city, with good and bad people of every kind out there, areas where some people are best advised not to go, certainly not a place you let young children wander around unsupervised, and for the most part you're on your own with any decision you make.

      The problem isn't the nature of the city, it's that some idiots think everyone should live in a theme park because they can't supervise their own kids.

      reply to this | link to this | view in chronology ]

  • identicon
    bob, 13 Aug 2020 @ 9:26pm

    testing.

    At least they had the fore knowledge to test the system with a teen before releasing it to the children of the world.

    reply to this | link to this | view in chronology ]

    • icon
      Scary Devil Monastery (profile), 14 Aug 2020 @ 1:09am

      Re: testing.

      True that. Can you imagine how triggerhappy the average obscenity filter would get once a few ten thousand 14 year olds had written their own version of what the long-necked giraffe wanted to do to the fluffy white bunny? I can envision now, the average business mail going;

      "Dear [REDACTED]. We are [REDACTED] to [REDACTED] your [REDACTED]..."

      reply to this | link to this | view in chronology ]

  • icon
    Scary Devil Monastery (profile), 14 Aug 2020 @ 12:57am

    Will they rediscover fire and the wheel next?

    ...because the hypothesis "We can stop teenagers from being lewd" stands debunked as a logical impossibility in ancient greek and roman memoirs, and probably on cuneiform clay tablets.

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Close

Add A Reply

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Follow Techdirt
Essential Reading
Techdirt Insider Chat
Recent Stories

Close

Email This

This feature is only available to registered users. Register or sign in to use it.