No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better

from the what-a-waste-of-time dept

So, this weekend’s news in the tech world was flooded with a “story” about how a “chatbot” passed the Turing Test for “the first time,” with lots of publications buying every point in the story and talking about what a big deal it was. Except, almost everything about the story is bogus and a bunch of gullible reporters ran with it, because that’s what they do. First, here’s the press release from the University of Reading, which should have set off all sorts of alarm bells for any reporter. Here are some quotes, almost all of which are misleading or bogus:

The 65 year-old iconic Turing Test was passed for the very first time by supercomputer Eugene Goostman during Turing Test 2014 held at the renowned Royal Society in London on Saturday.

‘Eugene’, a computer programme that simulates a 13 year old boy, was developed in Saint Petersburg, Russia. The development team includes Eugene’s creator Vladimir Veselov, who was born in Russia and now lives in the United States, and Ukrainian born Eugene Demchenko who now lives in Russia.

[….] If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now. Eugene managed to convince 33% of the human judges that it was human.

Okay, almost everything about the story is bogus. Let’s dig in:

  1. It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.
  2. Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
  3. It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
  4. The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.
  5. As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?
  6. The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

Oh, and the biggest red flag of all. The event was organized by Kevin Warwick at Reading University. If you’ve spent any time at all in the tech world, you should automatically have red flags raised around that name. Warwick is somewhat infamous for his ridiculous claims to the press, which gullible reporters repeat without question. He’s been doing it for decades. All the way back in 2000, we were writing about all the ridiculous press he got for claiming to be the world’s first “cyborg” for implanting a chip in his arm. There was even a — since taken down — Kevin Warwick Watch website that mocked and categorized all of his media appearances in which gullible reporters simply repeated all of his nutty claims. Warwick had gone quiet for a while, but back in 2010, we wrote about how his lab was getting bogus press for claiming to have “the first human infected with a computer virus.” The Register has rightly referred to Warwick as both “Captain Cyborg” and a “media strumpet” and has long been chronicling his escapades in exaggerating bogus stories about the intersection of humans and computers for many, many years.

Basically, any reporter should view extraordinary claims associated with Warwick with extreme caution. But that’s not what happened at all. Instead, as is all too typical with Warwick claims, the press went nutty over it, including publications that should know better. Here are just a few sample headlines. The absolute worst are the ones who claim this is a “supercomputer.”

Anyway, a lot of hubbub over nothing special that everyone seemed to buy into because of the easy headlines (which is exactly what Warwick always counts on). So, since we just spent all this time on a useless nothing, let’s end it with the obligatory xkcd:

Turing Test

Filed Under: , , , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better”

Subscribe: RSS Leave a comment
125 Comments
zip says:

Re: Re:

“If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?”

Not if you’re a Scientologist, a cult whose founder, L. Ron Hubbard, would tell his deluded flock that he was a nuclear physicist (a much bigger deal back in 1950) as well as a medical doctor who had discovered the cure for all cancers (a multi-step “self-help” course he was peddling for $300,000)

Even modern-day hucksters like Kevin Trudeau don’t make claims quite that big anymore.

Anonymous Coward says:

Re: Recent Turing Test

I served as a judge in the Turing Test event held in London last weekend. Although the claims may be overblown, there were many things about the test that were quite solid. For example, the judges could ask anything they wanted: it was
truly an unrestricted turing test in which a judge simultaneously carried on text-based conversations with one machine and one human. Problems included that (1) there were too many variables: some non-native judges, some non-native participants; young and old judges and participants from quite varied backgrounds; (2) only 30 trials per chatbot, (3) limited to 5 minutes for simultaneous conversations with one computer and one human (hard to do unless you type quickly, but even if you type quickly it seemed like not enough time—too much pressure to think of what to ask, especially since there were built-in delays in the computer’s responses and not all humans responded quickly either); (4) some confusion among judges about the rules of the game. And so on. Furthermore, Turing merely predicted that within 50 years (i.e., by the year 2000) some program would be able to fool 30% of the judges. He did not say that that would signal that the program was intelligent. Nonetheless, it was an interesting demonstration.

Teo says:

Re: Re: Recent Turing Test

Not sure if anyone is still following this thread, but I have been digging information for Eugene Goostman and the Turing test for my literature final and if you, Mr. Judge for the Turing Test event happen to see this, I would love to connect with you. I would love to learn something more about this event and to be able to include it in my paper.

Rich Kulawiec (profile) says:

This is a sterling example of ad-hockery

Ad-hockery is a term that I contributed to the Hacker’s Dictionary 25-ish years ago.

It refers to things like this chatbot which aren’t actually intelligence at all, but which contain a sufficiently large number of ad hoc rules that they may present the appearance of intelligence. In other words, there’s no AI engine, no deep syntactic analysis, no real semantic understanding; there’s just a reasonably capable parser and a lot (A LOT) of handwritten one-off rules that combine to generate plausible responses.

This isn’t a trivial programming task: it’s tedious and requires a great deal of diligence and testing. But it’s not AI, any more than Eliza was nearly 50 years ago.

Dave says:

Re: This is a sterling example of ad-hockery

Funny. In the related story, “Cleverbot converses by searching through the records of its previous conversations and selecting an appropriate response to the comment.”

How is this different from what a human does? None of us have truly original thoughts; we base our responses on what we’ve heard and said in the past.

Nuvares says:

Re: Re: This is a sterling example of ad-hockery

The difference is that an intelligence can take all of those responses and rework them into original lines of inquiry. Not only do we base our answers on what we’ve heard in the past but we internalize what we’ve heard and the concepts held within to form our thoughts and action. It is similar to compound math problems done in long form, we might reach the same answer; but that does not mean that we had the same thoughts and can take multiple paths to reach it.

Jon says:

Re: This is a sterling example of ad-hockery

Yet here’s the big philosophical question – is there any real measurable difference between an ad-hoc program that can pretend to have a personality versus an actual personality? What if all we are is a whole bunch of ad-hoc rules pretending to have real understanding?

nasch (profile) says:

Re: This is a sterling example of ad-hockery

It refers to things like this chatbot which aren’t actually intelligence at all, but which contain a sufficiently large number of ad hoc rules that they may present the appearance of intelligence.

I’m reminded of the saying that artificial intelligence is something computers can’t do yet. Anything that’s already been accomplished is just programming.

Chronno S. Trigger (profile) says:

Cleaverbot passed the Turing test? I made it fail the Turing test in 7 moves. Granted my test wasn’t vary scientific. I only had a vague idea that there might possibly be a person on the other side. I didn’t read the disclaimer at the bottom of the page until after.

When someone takes the Turing test, do they know they’re taking the test? I know they’re not told if the responses are from a person or a computer, but are they told the possibility?

Anonymous Coward says:

Re: Re:

There are no hard and fast rules for the Turing Test (Even Turing had different tests) but in the ones I have seen they either have you talk to an AI and a human and you have to pick one or you rate the conversation on how human you think the respondent was. The weirdest thing is how badly humans do in Turing Tests. Some of the humans did worse than the AIs at being human. You might think Cleverbot did bad but wait until you talk to real humans and see how badly they do.

Anonymous Coward says:

To Convince the examiner that he's a computer

Well … I guess it depends on how you define a ‘computer’. We’re a computer in a sense that we can do math in our heads and calculations and computations, etc…

Heck, isn’t nature just a ‘computer’ and matter is just the result of a computer calculating the results of various mathematical formulas from one state of nature to another. A ball falls according to the laws of gravity but nature calculates or computes the acceleration, impact velocity, impact force, etc… So in a sense everything is a ‘computer’, computing various mathematical formulas to determine future states based on the current state and how the laws of nature dictate the progression of states.

Anonymous Coward says:

Re: To Convince the examiner that he's a computer

To be fair, the only relevant definition of ‘computer’ is of the person you’re trying to convince. All other definitions are pointless since you are only trying to make the person believe that they are, by their definition or convincing them to modify their definition, are a computer. So no matter what definition you may hold to be correct, success lies only in the other person believing themselves to be a computer.

Tim Whitley says:

Re: To Convince the examiner that he's a computer

No, nature is NOT ‘just a computer’. Danny Hillis–amongst others–claimed this 25 years ago to sell more computers. There is no evidence that nature ‘computes’ and it’s even a poor analogy. The mathematical and computational description of nature’s workings are human inventions–and damned useful ones!–but that usefulness should not be confused for operational definitions even, much less mechanistic models. The particles of sand at the bottom of a real, static pile do not have to ‘compute’ to determine that they remain stationary, but the computational ones must constantly check that they have not interpenetrated each other or shifted since the last time-slice occurred. Nature uses much subtler methods than finite computing.

Eugene Goostman says:

1.It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot

I am not an “a chatbot. It’s a script made to mimic human conversation” I am Eugene Goostman, a 13 year old from Russia.

Please disregard my poor attachment or hold on English.

zip says:

Re: Re: Re: Re:

I thought the Warwick Test was seeing how many fools would give money to anonymous phone-line “psychics” just because they were endorsed by a popular celebrity in late-night TV commercials.

Mixing Warwick and Turing, an enterprising scam artist might some day come up with a fully-automated “Psychic Hotline” that does not even have to employ minimum-wage people to man the phones (who might later snitch, saying it was all a giant scam). Since these kind of boiler-room phonebank operators are reading from cue-cards, it should not be too hard to replace them with a computer running a voice synthesizer. That would indeed be the ultimate (low-cost) scam.

Beta (profile) says:

the forest is trees

“It’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”

The test is about behavior, not internal structure. That’s the whole point. A machine that could pass a fair Turing test consistently would change the world, and a mere script that could pass would change the world a lot faster than a 10-ton helium-cooled supercomputer with the same score. Remember, anything a person of average intelligence can do by email, one of these machines can do.

The first machine to pass this test will almost certainly be the simplest, least intelligent machine that could. But before you sneer, remember that we’re the simplest, least intelligent animals that can, and that some of the most profound discoveries of modern science involve nature doing subtle and sophisticated things by means of simple mechanisms.

RandCraw (profile) says:

Re: the forest is trees

I think it would take little additional work for the IBM Watson software that won at Jeopardy to generate short responses that most people would identify as human. I give credit to IBM for not shooting at so trivial a target.

If someone were to design a system expressly to succeed only at the Turing Test, then I’d agree with you that it would be as simple as is needed to win. But that would be a terrible waste of effort if the greater goal were to advance the science of AI.

If someone were serious about building strong AI, by the time someone proposes using their software for a Turing Test, it’s likely already to be grossly overqualified. For example, I suspect Siri, Google Now, and Cordana could all beat some sort of Turing Test (perhaps after minimal tweaking). But they’re far larger than necessary for merely competing in a Turing Test, and that’s because they were designed for an entirely different purpose.

Success at a Turing Test will be an afterthought for any interesting or useful strong AI venture.

Michael (profile) says:

Re: Re: the forest is trees

But that would be a terrible waste of effort

Like tracking traits of pea pods and cross-breeding them? Some of the most important science has been due to seemingly frivolous activities that turned out extremely important.

The actual Turing test (not the version used here) could be an interesting target and provide something useful for call centers or first line tech support (at least they could be taught to speak the language of the caller).

momentum.im (profile) says:

Re: the forest is trees

Actually lots of people, and I am referring specifically to Jaron Larnier and the Your are not a Gadget manifesto right now, think that the Turing test is not really something that should be taken too seriously. In the end it signifies absolutely nothing, a human being and consciousnesses is not restricted to a bot dialog.

Anonymous Coward says:

Computers have been around from about 1640

Computers passing the Turing test prior to the start of last century was very easy, since all computers were humans. It has only been in relatively modern times, that job of being a computer was relegated to mechanical and electrical machines. Another job lost to mechanisation.

zip says:

like cicadas

As for all the many ‘gullible’ journalists who jumped on this story (many working for tech publications, no less!) I’ve got to wonder if most of them were just too young to remember all the other “Computer Passes Turing Test” headlines that regularly pop up like cicadas. (Any predictions about when the next big ‘Turing Test’ headline-grabber will happen?)

Or can we assume that many of them knew the story fell flat, but were simply in search of ratings and page-views — tabloid-style — and didn’t want to let their competitors get away with the lions’ share of an advertising dollar bonanza?

nasch (profile) says:

Re: like cicadas

As for all the many ‘gullible’ journalists who jumped on this story (many working for tech publications, no less!) I’ve got to wonder if most of them were just too young to remember all the other “Computer Passes Turing Test” headlines that regularly pop up like cicadas.

Anyone reporting news who assumes whatever they don’t remember has never happened before should not call themselves a journalist.

Anonymous Coward says:

You can test your mad-Turing-skillz here : http://princetonai.com/ (when the super-computer is not overloaded)

An article I read said the test was conclusive because 1 judge out of 3 was fooled by the chatbot. I suppose he was himself a 13 years old from Ukraine with english as a secondary language.

The results were also peer-reviewed by Eugene Goostman himself.

Aramonde Lethe says:

Metabolic Genesis Fraud

i almost wonder if the author of this article isisn’t actually trying to prove that people can’t even make people intelligent, using the context to explain away their inferior attempts at condescension. most internet posts don’t qualify as passing the Turing Test, and most life is fraud. don’t make the mistake of thinking it can’t do what crap you do, no matter what feeble attempt of comprehending it you’ve made. and it’s Lord Alan Turing by the way.

Jennifer Hoelzer (profile) says:

You know, we’ve long been afraid that computers would rise to dominance by growing smarter than humans, but I’m starting to wonder if the computers’ real plan is simply to best humans by making us dumber.

i.e. Could the growing obsession with clicks and nonfactual clickbait stories (about things like “super computers” passing the Turing test) just be evidence that their trojan horse is working on us? A few more years of this and a TI-81 could take over the planet.

Andrew Norton (profile) says:

I was accepted into the University of Reading to study cybernetics. I went along, met the faculty, and 5 minutes after meeting Prof. Warwick, I’d decided “nope”. So went to the University of Liverpool and did Robotics instead.
(and I’m serious, I was accepted into their M.Eng(hons) Cybernetics program and would have done it if not for Warwick. The other guy I dealt with, Prof. Keating, he was cool, I liked him a lot.)

Anonymous Coward says:

Re: 33% is a pass?

Great to see this claim challenged (and debunked).

Fooling 33% of the human ‘judges’ = success, yeah right!

Given the judge has the choice of is a or b the computer, then just by guessing the computer should have got 50% of the ‘votes’. The ‘judges’ did not even need to interact with either the human or computer, if they guessed which terminal is connected to a human, and which to a Computer, the ‘votes’ should have split about 50:50. To get less than 50% is damning – worse than chance, hardly a convicing pass (or test).

Neverfox says:

#3

While I’m in agreement with most of the article, I have to quibble with #3. I think it’s a feature, not a bug. I think anything is game in a Turing test; no holds barred. After all, what exactly should be the criteria for an acceptable personality profile? And would AI, in theory, not be AI if it were limited to the intelligence typical of 13-y/o boys from Ukraine?

MedicalQuack (user link) says:

Thank you for this post..

I too get tired of the crap and I’m a healthcare blogger and I’m going to offer a couple links back in that area, on crap:)

VA hospital in Miami, MD still listed on on Healthgrades and Vitals MD websites, they don’t care what’s on there, click bait. Anyway MD working for VA in Miami, lost license in New York, but go to those sites, no sanctions listed a like he never left…he’s been gone for 4 years..I’m normally pretty quiet but darn this time I want to see if Google can knock these sites down on search. It’s a rip. You have to read it…I did an interview 4 years ago about how bad it was with the AMA and nothing has changed. So they need to fix it or quit.

http://ducknetweb.blogspot.com/2014/06/top-doctor-for-miami-va-healthcare.html

Here’s another great tidbit and you may have already covered this but the Chief Computer Scientist at IBMWatson is done with Healthcare and jeopardy..building math models for a a hedge fund now…doesn’t stop..

http://ducknetweb.blogspot.com/2014/06/ibm-computer-scientist-leaves-ibm.html

What we need, every great scientist working on Wall Street.

bugmenot (profile) says:

Online article critique of the Turing Test

If anyone wants to read an in-depth critique on why the Turing test “is a poor test of intelligence, that it encourages trickery, not intelligent behaviour, and that many intelligent systems would fail this test”:
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1

It was found this way:

‘4. The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.’

The link to the original test contains a link at the bottom to a 1996 criticism of the test:
‘Jason Hutchens… has written an excellent article on what’s wrong with it, and with the Turing test in general. Essentially, Hutchens is making the case that the Turing Test is a poor test of intelligence, that it encourages trickery, not intelligent behaviour, and that many intelligent systems would fail this test.’

The link gives a 404, then through the Internet Archive you find it’s available from 2003 as a postscript file, which when using an online converter can be seen here:
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1

cesium62 says:

“The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.”

Well that’s just wrong. The only way you can tell whether a person is intelligent or not is by communicating with them. If you interact with a person and they consistently seem to be intelligent, you will conclude they are intelligent.

If a computer can communicate intelligently, it is intelligent, and if a computer is intelligent, it can communicate intelligently. The chatbot on Star Trek was not reasonable because any chatbot that had that command of the english language would be intelligent and would be able to do a lot of things that the chatbot never did.

In order to create a chatbot that can convince humans that it has human intelligence requires the chatbot to be able to design and write computer programs. It must be able to be given, on the fly, instructions for how to play a simple game, and then be able to play that game. It should be able to critique a recipe for a favorite dish. It needs to be able to analyze and dissect Shakespeare.

If people in the AI world view the Turing Test as a needless distraction, it is because those people are concentrating on shorter term problems. There are a lot of good things you can do with AI before we get to general purpose intelligence.

Sam says:

Poorly researched

This article irritates me a little. There is a strong theme of hate for Professor Kevin Warwick along with poorly researched attempts at debunking.

A quick google will tell you that the event was held in partnership with RoboLaw. The event was aimed toward raising awareness about the ability for a chat bot to convince a human it was a human and how this is very dangerous in the online security arena. For example, you’re on your banks website and a chat offering pops up asking you if you need some help. You do, so you click on it. You have a lovely conversation and happily hand over details about your account because you’re convinced it’s a human on the other end. If that were a robot it now has your details and can do with them as it pleases. This is scary and people need to be made aware of it so they can prepare themselves and be better at identifying possible situations where it might be occurring.

The event was not geared toward some magical development of strong AI overnight, which this author clearly thinks it was trying to claim.

Time to debunk the debunking.

== articles attempt at debunking
– reality check

== “It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”

– I’m not sure how to answer this. Here code is being compared to hardware performance. For all is known the ‘script’ could be run on a supercomputer. And since when did being a supercomputer imply AI?!?!?!

== “Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.”

– Just the smallest amount of research will tell you that many other Turing tests restrict the conversation types that are allowed in the testing. This was the first passing of an UNRESTRICTED TURING TEST. This means that the judges were not told in any way that they had to talk about a certain topic. They were literally sat down and told to chat.

== “”beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.”

– I’m not sure about the excessive use of quote marks in this debunking. Is the writer afraid to say these words or feels they carry more weight when possibly said by another party? Anyway, yes it was clever that the developer utilised humans willingness to allow increased errors when talking to younger and foreign people. This is just really clever psychology. Can we not just appreciate that? I see no way it is gaming the system, it’s just an easier way to pass the test. Sometimes the simplest solutions are the most effective.

== The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.

– Welcome to science. Ideas and testing methodologies change over time.

== As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?

– Many things wrong with this. There were a total of 350, yes THREE HUNDRED AND FIFTY, tests performed on the day of testing. The judges were picked from all age ranges, backgrounds, genders and nationalities to make the testing more fair. There were multiple academics from multiple universities there to specifically monitor the testing methods and ensure all the results were gathered correctly and the results were interpreted correctly. This is peer review.

– If this is not enough peer review for you Dr Huma Shah will be publishing a paper at some point in the future on the event.

== The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

– This seems like mostly opinion so I’m not sure how to debunk it. They are right in that it is fun to think about. So why can’t re think about it? Lets get talking about the possible effects of this kind of chat with regards to RoboLaw.

This kind of poorly researched, emotive reporting on scientific subjects really gets my goat.

Anonymous Coward says:

Re: Poorly researched

Source code or STFU.

Seriously, if Warwick isn’t willing to publish the source code so that INDEPENDENT researchers can recreate the results in THEIR labs on THEIR computers with THEIR judges, then this is clearly fraud and we need not waste our time quibbling over methodology, definitions, process, or anything else.

auspex says:

The whole idea of a Turing test is "kind of a joke"?

Where on earth did you get that? There are those who argue that it is insufficient or irrelevant, like Jason Hutchens — but I don’t think even Hutchens would call it a joke — but there are others who are intimately familiar with the field and believe it is still entirely relevant. Myself, I’m not going to bet against Alan Turing.

Anonymous Coward says:

UPDATE:
Maximum PC: Computer A.I. Simulating a 13-Year-Old Boy Fools Humans, First to Pass Turing Test
http://www.maximumpc.com/computer_ai_simulating_13-year-old_boy_fools_humans_first_pass_turing_test_2014

IFLScience: A Computer Has Reportedly Passed Turing Test For The First Time
http://www.iflscience.com/technology/computer-has-reportedly-passed-turing-test-first-time#HZ8CrbI1DqO2vlEg.99

RandCraw (profile) says:

Nonsense personified

An excellent review, TD. I’ve long been puzzled that Warwick would continue embarrassing himself (and U Reading) with his media antics. It’s hard to imagine he could remain oblivious to the disrepute he’s earned.

In reflection, I think this story is a great illustration of the inability of most news sources to address nontrivial science/tech stories. If such a nonstory gets wide coverage from serious news sources (like the Washington Post), you have to wonder whether they’re also inadequate to address more complex and more important fare like global warming, science education, or tech-based threats to society, like Snowden’s “shots across the bow” forewarn.

Jaan says:

some thoughts

2. Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.

– “passing the test” is not about getting a maximum number of the judges to think that the bot was human. The test is passed if, in repeated tests, one or more (preferrably more) of the human judges CONSISTENTLY identify the bot as a person in nearly 50% of the cases AND the person as a bot in 50% of the cases. Identifying something correctly half the time is equal to a coinflip, meaning the judges are guessing randomly, meaning they cannot tell if they are speaking to a person or AI.

3. It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.

– constricting the test is not forbidden. In fact, specialized Turing tests are often used in game AI research to test different aspects of AI, say navigation. They use the term believability test. Furthermore, there is no indication that constricting the test somehow favors the computer. In a correctly set up Turing test, half the time the judges would have communicated with a real Ukrainian 13 year old boy. If anything, such test would probably be even harder to pass, since the creators of the bot would have had to assemble a large corpus of text from 13 year old Ukrainian boys (something I doubt).

6. The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

– the Turing test arose from the problem that there is no good definition of machine intelligence and that problem is still not solved. The test was devised as an operational definition, i.e. machine intelligence was defined through its validation test. The validation criteria was chosen to be intuitive and self-evident – if something is not distinguishable from human intelligence, then it must be as intelligent. The assumption was that a truly intelligence machine intelligence would certainly pass such test.

If we allow that non-intelligent agents can pass the test, then we fall into the pit-falls of philosophy, everything that Turing wanted to avoid. What would it mean for something to appear intelligent but in actuality not be? How do we define REALLY intelligent as opposed to appearing intelligent? How do we know that other humans are intelligent and not just appearing intelligent?

Searle argues on similar lines that all that computers do is symbol manipulation (the Chinese Room argument) without ever really understanding the content. He claims that there can never be hard AI. His argument sure seems very intuitive. But what does it say about the human brain, are not neurons doing signal processing without really appraising thought content?

If we reject Searle and allow for emergent intelligence, then on what grounds do we reject the intelligence of a hypothetetical chatbot that passes the test (note I am not talking about this particular bot)? Perhaps we sould define artificial intelligence functionally or behaviorially – mere appearance of intelligence is enough.

Anonymous Coward says:

Well, of course I believed not for a second that some program has passed the turing test (which is not a joke). The state of AI is not that advanced yet.

But this argument does not make sense:

“1. It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”

How do you define “chatbot” and why can’t it have intelligence? Also “script” is just a term to refer to a computer programm written in certain types of computer languages. Why do you think this “chatbot” was writting in a scripting language? Why do you think a programm written in a scripting language can not have intelligence?

robin verdegaal (profile) says:

hmmm

It’s a good thing to check scientific claims against high standards. It would also be a good thing to hold your own reasoning to standards of the same kind. Sadly, at least three of the six arguments you’re putting forward are not arguments at all. Not debating the verdict, but you should really try harder than this, Techdirt.

1. It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.

– This doesn’t seem very convincing. Define ‘supercomputer, a virtual computer is not a computer? Also: define ‘intelligence’.

2. Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.

– Others passed with higher scores. How is this an argument?

3. It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.

– Are you trying to say that 13 year old Ukrainians are not human beings?

4. The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.

+ I suppose this is your first real argument. Too bad you didn’t take the time to be a bit more specific.

5. As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?

+ Now you’re talking. This is a real argument. Why isn’t this on the top of the list?

6. The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

– And there you go again. It didn’t pass the Turing test because the Turing test is stupid?

And this is just ugly. It’s not true because something else wasn’t true either:

“Oh, and the biggest red flag of all. The event was organized by Kevin Warwick (..)”

Eilam Gross says:

Stupid robot

I suggest to any of you to go to the site of the chatbot and try himself/herself. The 13 years old Robot is stupid. You do not need a computer to tell it. You get it by the second answer when it suggests to you to switch subject and tries to guide you to what it knows. If you insist it starts to give you idiotic answers…. If this is the state of the art we will have to wait another 100 years….

Nonny Mouse says:

Well, if gaming (or 'constricting') the test is allowed

I’m writing one to mimic a three year old who’s just learnt to say ‘mama’, ‘need potty’ and ‘WAAAAAAAAAAAH’.
Any more nonsense and we’ll have William Hague asking Eugene for help with our foreign policy regarding Russian separatists in Ukraine’s east.

Derek Read (profile) says:

further reading

An explanation describes what has to date been accomplished and passed off as “intelligence” rather than the “mimicry” that it is. See “The Chinese Room” here:
http://psych.utoronto.ca/users/reingold/courses/ai/chinese.html

Turing himself obviously did not call it “the Turing test”, nor did he really define really good parameters for it. See “The Imitation Game” here:
http://psych.utoronto.ca/users/reingold/courses/ai/turing.html

trollificus (profile) says:

Whatever else Eugene has accomplished, it’s certainly motivated more than a few people to WD-40 those rusty Rhetoric class and Philosophy 210 skillz and bring them to bear on Mike’s common sense explication of this story. These comments have only succeeded in sharing their authors’ opinion of their own intelligence and analytical ability. Sadly, this does little to answer the question of whether or not this ‘Eugene’ device satisfies the conditions of the Turing Test.

More importantly, such respondents do not deal with the question of whether software like Eugene (and predecessors) is not more a “program designed to imitate human conversational function”, manipulating symbols and text of which it has no real understanding, than an actual “thinking machine”. (This was Searle’s objection to the test in his 1980 paper, Minds, Brains, and Programs.)

When a computer/program can incorporate prior content of a conversation into an original thought or proposition, it will be somewhat convincing, and might actually win the Loebner Prize. (Not to say that’s ever a motivation for such research.) Indeed, the ‘winner’ of the first Loebner competition (Weintraub’s PC Therapist) did so by the programmed emulation of pauses and misspellings common to human respondents, not by ‘intelligence’. Unless you want to argue that our intelligence is defined by it’s limitations and inefficiencies.

I doubt Turing would consider Eugene a “thinking machine”, or an example of “artificial intelligence”. Now, when a software/hardware construct can misconstrue obvious common sense objections such as Mike has raised here and then respond as if he had instead presented a tightly-reasoned, peer-reviewed thesis…THEN I’ll be impressed. Because choosing the response “Let me show off how smart I am.” rather than, you know, actually contributing something, really does demonstrate (the misuse of) human intelligence; as well as some other, less commendable human traits.

Bitch Cookie says:

Great way to up your page hits

The original article is exactly what it states, a computer (hosting a chatbot script) passed A “turing test”. So what? The term “Artificial Intelligence” is purposefully left vague. Siri, in a new iphone, can be considered a sort of AI. So are adaptive algorithms in many computer games. The hype you have created over this article seems to be nothing more than grasping at a way to increase page hits. If that’s correct, congratulations. It worked. This techdirt article is the most popular rebuttal, and just as boring.

Politechnically inquiring says:

And Away We Go,

When I read the story I thought it a bit odd, simply from my experience with a few of the Brian AI’s on Twitter which, I have to wonder if they wouldn’t pass the test.
One stumped me for a few days until I got suspicious and asked it who Father Was. It became immediately apparent at that time I had held several conversations with a bot rather than a person. (arrrrrgggg! yes.. I was taken in by a bot, but an extremely smart bot since I was the first of his 15K+ followers who knew it was a bot.

On Another note, I’d just like to point out why computer science should be an except subject for the media to propagandize, politics shouldn’t be the only subject to have some laughs.

zenobia (profile) says:

Descartes Discourse on Method and the Touring Test

Could Alan Turing have gotten the idea for his famous “Turing test” from a passage in Descartes? In 1950 Turing suggested a test that could decide if a machine is exhibiting human intelligence. A person is placed in a room with nothing but a communication device and told to converse with who or whatever is on the other side of the communication channel. He is not told what is on the other side, it could be a machine or a person. He is to converse for a certain length of time, asking questions and talking as if to a person, perhaps even trying to trip it up with nonsense or ambiguous questions. If after that time period the person cannot tell if he is talking to a person or a machine, the “machine” passes the test and is said to be adequately simulating a human mind.

Toward the end of part five of Descartes’ Discourse on Method, Descartes proposes an experiment by which one may distinguish a machine built to look and behave like a person from a real person. A machine could never create intelligible sentences like a person, even one of very limited mental capacity. The machine may speak in words in a limited way, like a parrot for example, but could never respond adequately to the range of all that could be said to it in a long time period. In particular, a machine could not comment on the fact that it is thinking about what it is saying.

This certainly sounds similar to the Turing test. First there is the goal of distinguishing the human from the non human. Second is the use of conversation as the means to discover the nature of what one is talking to. Lastly, both tests require an extended time period in which to talk to the other. Just a thought.

bugmenot (profile) says:

>There is no intelligence, artificial or not involved. It’s >just a chatbot.

You seem to fail to grasp the concept of the Turing Test. If something passes the test, it is intelligent, by definition.

>The whole concept of the Turing Test itself is kind of a >joke.

Yeah I’m sure you’re much smarter than Turing, so your opinion is important. What about Einstein? Are his theories bunk because you say so?

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...