No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better

from the what-a-waste-of-time dept

So, this weekend's news in the tech world was flooded with a "story" about how a "chatbot" passed the Turing Test for "the first time," with lots of publications buying every point in the story and talking about what a big deal it was. Except, almost everything about the story is bogus and a bunch of gullible reporters ran with it, because that's what they do. First, here's the press release from the University of Reading, which should have set off all sorts of alarm bells for any reporter. Here are some quotes, almost all of which are misleading or bogus:
The 65 year-old iconic Turing Test was passed for the very first time by supercomputer Eugene Goostman during Turing Test 2014 held at the renowned Royal Society in London on Saturday.

'Eugene', a computer programme that simulates a 13 year old boy, was developed in Saint Petersburg, Russia. The development team includes Eugene's creator Vladimir Veselov, who was born in Russia and now lives in the United States, and Ukrainian born Eugene Demchenko who now lives in Russia.

[....] If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now. Eugene managed to convince 33% of the human judges that it was human.
Okay, almost everything about the story is bogus. Let's dig in:
  1. It's not a "supercomputer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot.
  2. Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
  3. It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
  4. The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.
  5. As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right?
  6. The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
Oh, and the biggest red flag of all. The event was organized by Kevin Warwick at Reading University. If you've spent any time at all in the tech world, you should automatically have red flags raised around that name. Warwick is somewhat infamous for his ridiculous claims to the press, which gullible reporters repeat without question. He's been doing it for decades. All the way back in 2000, we were writing about all the ridiculous press he got for claiming to be the world's first "cyborg" for implanting a chip in his arm. There was even a -- since taken down -- Kevin Warwick Watch website that mocked and categorized all of his media appearances in which gullible reporters simply repeated all of his nutty claims. Warwick had gone quiet for a while, but back in 2010, we wrote about how his lab was getting bogus press for claiming to have "the first human infected with a computer virus." The Register has rightly referred to Warwick as both "Captain Cyborg" and a "media strumpet" and has long been chronicling his escapades in exaggerating bogus stories about the intersection of humans and computers for many, many years.

Basically, any reporter should view extraordinary claims associated with Warwick with extreme caution. But that's not what happened at all. Instead, as is all too typical with Warwick claims, the press went nutty over it, including publications that should know better. Here are just a few sample headlines. The absolute worst are the ones who claim this is a "supercomputer." Anyway, a lot of hubbub over nothing special that everyone seemed to buy into because of the easy headlines (which is exactly what Warwick always counts on). So, since we just spent all this time on a useless nothing, let's end it with the obligatory xkcd:
Turing Test

Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. identicon
    Jaan, 10 Jun 2014 @ 9:30am

    some thoughts

    2. Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.

    - "passing the test" is not about getting a maximum number of the judges to think that the bot was human. The test is passed if, in repeated tests, one or more (preferrably more) of the human judges CONSISTENTLY identify the bot as a person in nearly 50% of the cases AND the person as a bot in 50% of the cases. Identifying something correctly half the time is equal to a coinflip, meaning the judges are guessing randomly, meaning they cannot tell if they are speaking to a person or AI.

    3. It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.

    - constricting the test is not forbidden. In fact, specialized Turing tests are often used in game AI research to test different aspects of AI, say navigation. They use the term believability test. Furthermore, there is no indication that constricting the test somehow favors the computer. In a correctly set up Turing test, half the time the judges would have communicated with a real Ukrainian 13 year old boy. If anything, such test would probably be even harder to pass, since the creators of the bot would have had to assemble a large corpus of text from 13 year old Ukrainian boys (something I doubt).

    6. The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

    - the Turing test arose from the problem that there is no good definition of machine intelligence and that problem is still not solved. The test was devised as an operational definition, i.e. machine intelligence was defined through its validation test. The validation criteria was chosen to be intuitive and self-evident - if something is not distinguishable from human intelligence, then it must be as intelligent. The assumption was that a truly intelligence machine intelligence would certainly pass such test.

    If we allow that non-intelligent agents can pass the test, then we fall into the pit-falls of philosophy, everything that Turing wanted to avoid. What would it mean for something to appear intelligent but in actuality not be? How do we define REALLY intelligent as opposed to appearing intelligent? How do we know that other humans are intelligent and not just appearing intelligent?

    Searle argues on similar lines that all that computers do is symbol manipulation (the Chinese Room argument) without ever really understanding the content. He claims that there can never be hard AI. His argument sure seems very intuitive. But what does it say about the human brain, are not neurons doing signal processing without really appraising thought content?

    If we reject Searle and allow for emergent intelligence, then on what grounds do we reject the intelligence of a hypothetetical chatbot that passes the test (note I am not talking about this particular bot)? Perhaps we sould define artificial intelligence functionally or behaviorially - mere appearance of intelligence is enough.

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: Copying Is Not Theft
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories

Close

Email This

This feature is only available to registered users. Register or sign in to use it.