No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better
from the what-a-waste-of-time dept
So, this weekend’s news in the tech world was flooded with a “story” about how a “chatbot” passed the Turing Test for “the first time,” with lots of publications buying every point in the story and talking about what a big deal it was. Except, almost everything about the story is bogus and a bunch of gullible reporters ran with it, because that’s what they do. First, here’s the press release from the University of Reading, which should have set off all sorts of alarm bells for any reporter. Here are some quotes, almost all of which are misleading or bogus:
The 65 year-old iconic Turing Test was passed for the very first time by supercomputer Eugene Goostman during Turing Test 2014 held at the renowned Royal Society in London on Saturday.
‘Eugene’, a computer programme that simulates a 13 year old boy, was developed in Saint Petersburg, Russia. The development team includes Eugene’s creator Vladimir Veselov, who was born in Russia and now lives in the United States, and Ukrainian born Eugene Demchenko who now lives in Russia.
[….] If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now. Eugene managed to convince 33% of the human judges that it was human.
Okay, almost everything about the story is bogus. Let’s dig in:
- It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.
- Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
- It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
- The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.
- As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?
- The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
Oh, and the biggest red flag of all. The event was organized by Kevin Warwick at Reading University. If you’ve spent any time at all in the tech world, you should automatically have red flags raised around that name. Warwick is somewhat infamous for his ridiculous claims to the press, which gullible reporters repeat without question. He’s been doing it for decades. All the way back in 2000, we were writing about all the ridiculous press he got for claiming to be the world’s first “cyborg” for implanting a chip in his arm. There was even a — since taken down — Kevin Warwick Watch website that mocked and categorized all of his media appearances in which gullible reporters simply repeated all of his nutty claims. Warwick had gone quiet for a while, but back in 2010, we wrote about how his lab was getting bogus press for claiming to have “the first human infected with a computer virus.” The Register has rightly referred to Warwick as both “Captain Cyborg” and a “media strumpet” and has long been chronicling his escapades in exaggerating bogus stories about the intersection of humans and computers for many, many years.
Basically, any reporter should view extraordinary claims associated with Warwick with extreme caution. But that’s not what happened at all. Instead, as is all too typical with Warwick claims, the press went nutty over it, including publications that should know better. Here are just a few sample headlines. The absolute worst are the ones who claim this is a “supercomputer.”
- The Verge: Computer passes Turing Test for first time by convincing judges it is a 13-year-old boy
- Venture Beat: Talk to the computer that passed the Turing Test, a historic artificial intelligence milestone
- Yahoo Tech: Turing Test Bested, Robot Overlords Creep Closer
- NBC News: Turing Test: Computer Program Convinces Judges It’s Human
- Washington Post: A computer just passed the Turing Test in landmark trial
- The Independent: Turing Test breakthrough as super-computer becomes first to convince us it’s human
- PC World: An AI milestone: Chatbot passes Turing Test by posing as 13-year-old boy
- The Wire: For the First Time Ever, a Computer Passed Turing Test for Artificial Intelligence
- Gizmodo: A Computer Program Has Passed the Turing Test For the First Time
- ZDNet: Computer chatbot ‘Eugene Goostman’ passes the Turing test
- Ars Technica: Eugene—the supercomputer, not 13-year-old—first to beat the Turing Test
- The Guardian: Computer simulating 13-year-old boy becomes first to pass Turing test
- CNET: Computer fools humans, passes Turing Test
- Computerworld: Supercomputer passes Turing Test by posing as a teenager
- Science Alert: Meet the first computer to pass the Turing Test
Anyway, a lot of hubbub over nothing special that everyone seemed to buy into because of the easy headlines (which is exactly what Warwick always counts on). So, since we just spent all this time on a useless nothing, let’s end it with the obligatory xkcd:
Filed Under: artificial intelligence, chatbot, eugene goostman, gullible journalists, hype, kevin warwick, supercomputer, turing test
Comments on “No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better”
If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?
Why? It would look great to have a fusion reactor on my bridge.
Re: Re:
As the Emperor learned, you really don’t want a reactor on your bridge, no matter how it lifts the decor.
Re: Re:
“If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?”
Not if you’re a Scientologist, a cult whose founder, L. Ron Hubbard, would tell his deluded flock that he was a nuclear physicist (a much bigger deal back in 1950) as well as a medical doctor who had discovered the cure for all cancers (a multi-step “self-help” course he was peddling for $300,000)
Even modern-day hucksters like Kevin Trudeau don’t make claims quite that big anymore.
Re: Re: Re:
I quit being a Scientologist.
It was great until Tom Cruise’s career was destroyed by the internet and dragged the entire religion down with it.
Stupid internet people.
Re: Re: Re: Re:
This must be a troll… You’re trolling, right?
Re: Re: Re:2 Re:
I do believe the word you’re looking for is “joking”, seeing as it was just a few days ago that we had an article on how the internet supposedly ruined Tom Cruise’s reputation over scientology.
Re: Re: Re:3 Re:
It is completely possible that Lars’ internet has been forced to comply with a request to forget that information.
Re: Re:
Actually, a fusion reactor is being built in France, at the Cadarache center, and it passed a series of tests before
Re: Re:
That you think it would be great to have a fusion reactor on your bridge, has nothing to do with what you’re replying to. The point is, you don’t just believe someone saying they created it, without actually seeing the proof that it works.
Re: dont waste your ur..time in fantasies
I m glad you like fusion, me too!|
Thanks for the background. I read the Ars Technica article yesterday evening and it seemed to be a pretty sketchy claim. Good to know I can pretty much write it off as complete bullshit without needing to waste more time investigating.
Re: Re:
Re: Recent Turing Test
I served as a judge in the Turing Test event held in London last weekend. Although the claims may be overblown, there were many things about the test that were quite solid. For example, the judges could ask anything they wanted: it was
truly an unrestricted turing test in which a judge simultaneously carried on text-based conversations with one machine and one human. Problems included that (1) there were too many variables: some non-native judges, some non-native participants; young and old judges and participants from quite varied backgrounds; (2) only 30 trials per chatbot, (3) limited to 5 minutes for simultaneous conversations with one computer and one human (hard to do unless you type quickly, but even if you type quickly it seemed like not enough time—too much pressure to think of what to ask, especially since there were built-in delays in the computer’s responses and not all humans responded quickly either); (4) some confusion among judges about the rules of the game. And so on. Furthermore, Turing merely predicted that within 50 years (i.e., by the year 2000) some program would be able to fool 30% of the judges. He did not say that that would signal that the program was intelligent. Nonetheless, it was an interesting demonstration.
Re: Re: Recent Turing Test
Not sure if anyone is still following this thread, but I have been digging information for Eugene Goostman and the Turing test for my literature final and if you, Mr. Judge for the Turing Test event happen to see this, I would love to connect with you. I would love to learn something more about this event and to be able to include it in my paper.
Well that escalated quickly
Computers are not capable of consciousness or higher order intelligence and thinking and deceiving. It’s absurd. We, uh, THEY are just machines.
Re: Re:
Careful with that thinking….Singularity is predicted around 2040-2060 and then we will see I suppose. Google “technological singularity”
Re: Re: Re:
or “Google technological singularity”.
This is a sterling example of ad-hockery
Ad-hockery is a term that I contributed to the Hacker’s Dictionary 25-ish years ago.
It refers to things like this chatbot which aren’t actually intelligence at all, but which contain a sufficiently large number of ad hoc rules that they may present the appearance of intelligence. In other words, there’s no AI engine, no deep syntactic analysis, no real semantic understanding; there’s just a reasonably capable parser and a lot (A LOT) of handwritten one-off rules that combine to generate plausible responses.
This isn’t a trivial programming task: it’s tedious and requires a great deal of diligence and testing. But it’s not AI, any more than Eliza was nearly 50 years ago.
Re: This is a sterling example of ad-hockery
Funny. In the related story, “Cleverbot converses by searching through the records of its previous conversations and selecting an appropriate response to the comment.”
How is this different from what a human does? None of us have truly original thoughts; we base our responses on what we’ve heard and said in the past.
Re: Re: This is a sterling example of ad-hockery
The difference is that an intelligence can take all of those responses and rework them into original lines of inquiry. Not only do we base our answers on what we’ve heard in the past but we internalize what we’ve heard and the concepts held within to form our thoughts and action. It is similar to compound math problems done in long form, we might reach the same answer; but that does not mean that we had the same thoughts and can take multiple paths to reach it.
Re: Re: This is a sterling example of ad-hockery
That’s okay, just tell me what city you live in.
Re: This is a sterling example of ad-hockery
Yet here’s the big philosophical question – is there any real measurable difference between an ad-hoc program that can pretend to have a personality versus an actual personality? What if all we are is a whole bunch of ad-hoc rules pretending to have real understanding?
Re: Re: This is a sterling example of ad-hockery
No; someone would have to have made up the rules, and if all we are is a collection of rules, then we couldn’t have done it.
Re: This is a sterling example of ad-hockery
I’m not sure it even rises to that level. In this instance, and especially with Warwick involved, feels more like “sad-hokery”.
Re: This is a sterling example of ad-hockery
“Real semantic understanding?” That’s the holy grail of AI, is it not? Never been done and never gonna be done computationally. And you’re right – even as a Chinese room, this is pathetic.
Re: This is a sterling example of ad-hockery
It refers to things like this chatbot which aren’t actually intelligence at all, but which contain a sufficiently large number of ad hoc rules that they may present the appearance of intelligence.
I’m reminded of the saying that artificial intelligence is something computers can’t do yet. Anything that’s already been accomplished is just programming.
Cleaverbot passed the Turing test? I made it fail the Turing test in 7 moves. Granted my test wasn’t vary scientific. I only had a vague idea that there might possibly be a person on the other side. I didn’t read the disclaimer at the bottom of the page until after.
When someone takes the Turing test, do they know they’re taking the test? I know they’re not told if the responses are from a person or a computer, but are they told the possibility?
Re: Re:
Some of the shills around here seem to fail the turing test. I can’t tell if I’m talking to a bot or a retarded shill.
Re: Re:
The original test featured a human chatting with a computer, and another person had to judge which participant was which.
Re: Re:
There are no hard and fast rules for the Turing Test (Even Turing had different tests) but in the ones I have seen they either have you talk to an AI and a human and you have to pick one or you rate the conversation on how human you think the respondent was. The weirdest thing is how badly humans do in Turing Tests. Some of the humans did worse than the AIs at being human. You might think Cleverbot did bad but wait until you talk to real humans and see how badly they do.
Re: Re: Re:
The weirdest thing is how badly humans do in Turing Tests.
I dunno if that’s weird; I’ve read countless posts supposedly by humans that I’ve wished hoped and prayed were created by something inhuman.
Re: Re: Re:
The Turing Test is pretty much bullshit. Turing proposed it in a highly speculative manner, and the idea has a TON of problems. One of them being that many human fail it.
Re: Re:
As far as i know the Turing Test needs 3 elements : (1)a computer chatting with (2)a human, and (3)human observers who only see the dialogue in written form. The test is passed if 30% or more of the observers can’t tell the man apart from the computer in a series of 5 minutes chats.
To Convince the examiner that he's a computer
Well … I guess it depends on how you define a ‘computer’. We’re a computer in a sense that we can do math in our heads and calculations and computations, etc…
Heck, isn’t nature just a ‘computer’ and matter is just the result of a computer calculating the results of various mathematical formulas from one state of nature to another. A ball falls according to the laws of gravity but nature calculates or computes the acceleration, impact velocity, impact force, etc… So in a sense everything is a ‘computer’, computing various mathematical formulas to determine future states based on the current state and how the laws of nature dictate the progression of states.
Re: To Convince the examiner that he's a computer
To be fair, the only relevant definition of ‘computer’ is of the person you’re trying to convince. All other definitions are pointless since you are only trying to make the person believe that they are, by their definition or convincing them to modify their definition, are a computer. So no matter what definition you may hold to be correct, success lies only in the other person believing themselves to be a computer.
Re: To Convince the examiner that he's a computer
You’re right in questioning the definition of ‘computer’
‘computer’ was actually widely used as a job title before it referred to a machine (mechanical or electrical)
Re: To Convince the examiner that he's a computer
No, nature isn’t a computer; it would need to have hardware that works, an operating system, programs and interfaces. Nature could not have created those things, someone else must have been the “programmer.”
Re: Re: To Convince the examiner that he's a computer
Nature has all those things. A “programmer” in the sense of an outsider who arranges everything is unnecessary.
Re: To Convince the examiner that he's a computer
No, nature is NOT ‘just a computer’. Danny Hillis–amongst others–claimed this 25 years ago to sell more computers. There is no evidence that nature ‘computes’ and it’s even a poor analogy. The mathematical and computational description of nature’s workings are human inventions–and damned useful ones!–but that usefulness should not be confused for operational definitions even, much less mechanistic models. The particles of sand at the bottom of a real, static pile do not have to ‘compute’ to determine that they remain stationary, but the computational ones must constantly check that they have not interpenetrated each other or shifted since the last time-slice occurred. Nature uses much subtler methods than finite computing.
1.It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot
I am not an “a chatbot. It’s a script made to mimic human conversation” I am Eugene Goostman, a 13 year old from Russia.
Please disregard my poor attachment or hold on English.
Re: Re:
He’s not from Russia…yet.
This is exactly what the skynet alpha would want you to believe!
CNET couldn’t even get Warwick’s name right.
In fact, perhaps we have all been fooled. I think it is possible that some of these “news sources” could be chatbots.
Re: Re:
Nah, the level of apparent intelligence would have been higher.
don't be silly!
>>I think it is possible that some of these “news sources”
>>could be chatbots.
That’s ridiculous, when the chatbots are showing so much more intelligence!
The new Turing Test is to reduce the bar of intelligence of the persons you are trying to convince below that of the chatbot/script that is trying to do the convincing, rather than trying to raise the intelligence of the chatbot/script to that of human standards.
But what do you call the test to determine if a hack like Warwick succeeds in fooling 90% of the media into believing he is a credible news source?
Re: Re:
The Warwick test. Successfully convince people that a machine convinced people that it was a person.
Re: Re: Re:
I thought the Warwick test was just directions to San Jose.
Re: Re: Re: Re:
I thought the Warwick Test was seeing how many fools would give money to anonymous phone-line “psychics” just because they were endorsed by a popular celebrity in late-night TV commercials.
Mixing Warwick and Turing, an enterprising scam artist might some day come up with a fully-automated “Psychic Hotline” that does not even have to employ minimum-wage people to man the phones (who might later snitch, saying it was all a giant scam). Since these kind of boiler-room phonebank operators are reading from cue-cards, it should not be too hard to replace them with a computer running a voice synthesizer. That would indeed be the ultimate (low-cost) scam.
Re: Re:
The Darwin Test?
the forest is trees
“It’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”
The test is about behavior, not internal structure. That’s the whole point. A machine that could pass a fair Turing test consistently would change the world, and a mere script that could pass would change the world a lot faster than a 10-ton helium-cooled supercomputer with the same score. Remember, anything a person of average intelligence can do by email, one of these machines can do.
The first machine to pass this test will almost certainly be the simplest, least intelligent machine that could. But before you sneer, remember that we’re the simplest, least intelligent animals that can, and that some of the most profound discoveries of modern science involve nature doing subtle and sophisticated things by means of simple mechanisms.
Re: the forest is trees
I think it would take little additional work for the IBM Watson software that won at Jeopardy to generate short responses that most people would identify as human. I give credit to IBM for not shooting at so trivial a target.
If someone were to design a system expressly to succeed only at the Turing Test, then I’d agree with you that it would be as simple as is needed to win. But that would be a terrible waste of effort if the greater goal were to advance the science of AI.
If someone were serious about building strong AI, by the time someone proposes using their software for a Turing Test, it’s likely already to be grossly overqualified. For example, I suspect Siri, Google Now, and Cordana could all beat some sort of Turing Test (perhaps after minimal tweaking). But they’re far larger than necessary for merely competing in a Turing Test, and that’s because they were designed for an entirely different purpose.
Success at a Turing Test will be an afterthought for any interesting or useful strong AI venture.
Re: Re: the forest is trees
But that would be a terrible waste of effort
Like tracking traits of pea pods and cross-breeding them? Some of the most important science has been due to seemingly frivolous activities that turned out extremely important.
The actual Turing test (not the version used here) could be an interesting target and provide something useful for call centers or first line tech support (at least they could be taught to speak the language of the caller).
Re: the forest is trees
Actually lots of people, and I am referring specifically to Jaron Larnier and the Your are not a Gadget manifesto right now, think that the Turing test is not really something that should be taken too seriously. In the end it signifies absolutely nothing, a human being and consciousnesses is not restricted to a bot dialog.
Hmmm, convince the world that you are a sullen 13 year old boy with poor English skills… “boob”.
Computers have been around from about 1640
Computers passing the Turing test prior to the start of last century was very easy, since all computers were humans. It has only been in relatively modern times, that job of being a computer was relegated to mechanical and electrical machines. Another job lost to mechanisation.
like cicadas
As for all the many ‘gullible’ journalists who jumped on this story (many working for tech publications, no less!) I’ve got to wonder if most of them were just too young to remember all the other “Computer Passes Turing Test” headlines that regularly pop up like cicadas. (Any predictions about when the next big ‘Turing Test’ headline-grabber will happen?)
Or can we assume that many of them knew the story fell flat, but were simply in search of ratings and page-views — tabloid-style — and didn’t want to let their competitors get away with the lions’ share of an advertising dollar bonanza?
Re: like cicadas
As for all the many ‘gullible’ journalists who jumped on this story (many working for tech publications, no less!) I’ve got to wonder if most of them were just too young to remember all the other “Computer Passes Turing Test” headlines that regularly pop up like cicadas.
Anyone reporting news who assumes whatever they don’t remember has never happened before should not call themselves a journalist.
You can test your mad-Turing-skillz here : http://princetonai.com/ (when the super-computer is not overloaded)
An article I read said the test was conclusive because 1 judge out of 3 was fooled by the chatbot. I suppose he was himself a 13 years old from Ukraine with english as a secondary language.
The results were also peer-reviewed by Eugene Goostman himself.
Metabolic Genesis Fraud
i almost wonder if the author of this article isisn’t actually trying to prove that people can’t even make people intelligent, using the context to explain away their inferior attempts at condescension. most internet posts don’t qualify as passing the Turing Test, and most life is fraud. don’t make the mistake of thinking it can’t do what crap you do, no matter what feeble attempt of comprehending it you’ve made. and it’s Lord Alan Turing by the way.
Re: Metabolic Genesis Fraud
When you make a silly goal, like the turing test, then you get silly people.
Chat bots are expensive, I’d rather just buy some third worlders to upvote my reddit posts.
You know, we’ve long been afraid that computers would rise to dominance by growing smarter than humans, but I’m starting to wonder if the computers’ real plan is simply to best humans by making us dumber.
i.e. Could the growing obsession with clicks and nonfactual clickbait stories (about things like “super computers” passing the Turing test) just be evidence that their trojan horse is working on us? A few more years of this and a TI-81 could take over the planet.
Re: Re:
Don’t worry. We cannot get any dumber.
Sounds Like The Common Core
…passed the multiple choice test.
….learned nothing.
No intelligence? Really?
“There is no intelligence, artificial or not involved.”
First point on the list and the critic is already disqualifying themselves by demonstrating that they don’t even understand the notion of intelligence.
Re: No intelligence? Really?
/* First point on the list and the critic is already disqualifying themselves by demonstrating that they don’t even understand the notion of intelligence. */
… well, to be fair to him, no one really does.
This is all a meandering postscript to Dennett
Seriously. Read Dennett’s fun, short 1985 paper: “Can Machines Think?” There’s tons of .pdfs of it around, and it deals with the issue thoroughly.
That paper should be required reading for journalists who report on the Turing test.
Thanks
for the truth
Re: Thanks
I was accepted into the University of Reading to study cybernetics. I went along, met the faculty, and 5 minutes after meeting Prof. Warwick, I’d decided “nope”. So went to the University of Liverpool and did Robotics instead.
(and I’m serious, I was accepted into their M.Eng(hons) Cybernetics program and would have done it if not for Warwick. The other guy I dealt with, Prof. Keating, he was cool, I liked him a lot.)
33% is a pass? In my book book that’s a fail, pure and simple. I just had a thought; this could mean that James Clapper is a chatbot because it keeps spouting the same crap over and over and until it can prove that it’s human then he’s a NSA chatbot.
Re: 33% is a pass?
Great to see this claim challenged (and debunked).
Fooling 33% of the human ‘judges’ = success, yeah right!
Given the judge has the choice of is a or b the computer, then just by guessing the computer should have got 50% of the ‘votes’. The ‘judges’ did not even need to interact with either the human or computer, if they guessed which terminal is connected to a human, and which to a Computer, the ‘votes’ should have split about 50:50. To get less than 50% is damning – worse than chance, hardly a convicing pass (or test).
#3
While I’m in agreement with most of the article, I have to quibble with #3. I think it’s a feature, not a bug. I think anything is game in a Turing test; no holds barred. After all, what exactly should be the criteria for an acceptable personality profile? And would AI, in theory, not be AI if it were limited to the intelligence typical of 13-y/o boys from Ukraine?
Re: #3
Anything is game? Awesome. I just wrote a chatbot that simulates a human in a persistent vegetative coma. So far in tests 50% of judges failed to pick it out from a real human in a persistent vegetative coma – that’s no better than just guessing! Quick, call Kevin Warwick!
Eugene who? University of what?....you'd be better off chatting with my cat (link to bot)
Ask Poppy: what do you think about artificial intelligence?
Great work
Thanks for the heads up. We need lots more skeptics like you these days.
I’d like to see @Techdirt have a ‘skeptics’ link for all news that comes out. A Mythbusters of tech news.
Alternative Turing test
Great to see al the critical articles popping up! I’ve written a post in which I propose an alternative Turing test for Augmented Intelligence: http://www.mindextensions.org/articles/turing-test-for-extended-cognition/
Re: Alternative Turing test
So if I’m reading this correctly… you’re for just giving up on the whole simulating-human-intelligence thing, huh?
/* Many in the AI world look on the Turing Test as a needless distraction. */
Item 1: [citation needed]
Item 2: A distraction it might very well be, but it’s the only way of ducking a very real problem in both information science and psychology: how exactly does one quantify “intelligence,” anyway?
Thank you for this post..
I too get tired of the crap and I’m a healthcare blogger and I’m going to offer a couple links back in that area, on crap:)
VA hospital in Miami, MD still listed on on Healthgrades and Vitals MD websites, they don’t care what’s on there, click bait. Anyway MD working for VA in Miami, lost license in New York, but go to those sites, no sanctions listed a like he never left…he’s been gone for 4 years..I’m normally pretty quiet but darn this time I want to see if Google can knock these sites down on search. It’s a rip. You have to read it…I did an interview 4 years ago about how bad it was with the AMA and nothing has changed. So they need to fix it or quit.
http://ducknetweb.blogspot.com/2014/06/top-doctor-for-miami-va-healthcare.html
Here’s another great tidbit and you may have already covered this but the Chief Computer Scientist at IBMWatson is done with Healthcare and jeopardy..building math models for a a hedge fund now…doesn’t stop..
http://ducknetweb.blogspot.com/2014/06/ibm-computer-scientist-leaves-ibm.html
What we need, every great scientist working on Wall Street.
Re: Thank you for this post..
thank for a post exemplifying the difficulties of using scripts to emulate human-generated content
Shocked Information
Really? Hows that possible. So, now chat bots will be online rather than being a real human? It’s really not worth. But a great achievement in the field of Computer Science.
Online article critique of the Turing Test
If anyone wants to read an in-depth critique on why the Turing test “is a poor test of intelligence, that it encourages trickery, not intelligent behaviour, and that many intelligent systems would fail this test”:
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1
It was found this way:
‘4. The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.’
The link to the original test contains a link at the bottom to a 1996 criticism of the test:
‘Jason Hutchens… has written an excellent article on what’s wrong with it, and with the Turing test in general. Essentially, Hutchens is making the case that the Turing Test is a poor test of intelligence, that it encourages trickery, not intelligent behaviour, and that many intelligent systems would fail this test.’
The link gives a 404, then through the Internet Archive you find it’s available from 2003 as a postscript file, which when using an online converter can be seen here:
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1
“The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.”
Well that’s just wrong. The only way you can tell whether a person is intelligent or not is by communicating with them. If you interact with a person and they consistently seem to be intelligent, you will conclude they are intelligent.
If a computer can communicate intelligently, it is intelligent, and if a computer is intelligent, it can communicate intelligently. The chatbot on Star Trek was not reasonable because any chatbot that had that command of the english language would be intelligent and would be able to do a lot of things that the chatbot never did.
In order to create a chatbot that can convince humans that it has human intelligence requires the chatbot to be able to design and write computer programs. It must be able to be given, on the fly, instructions for how to play a simple game, and then be able to play that game. It should be able to critique a recipe for a favorite dish. It needs to be able to analyze and dissect Shakespeare.
If people in the AI world view the Turing Test as a needless distraction, it is because those people are concentrating on shorter term problems. There are a lot of good things you can do with AI before we get to general purpose intelligence.
Re: Re:
While I agree on the point that strong AI will be displayed when a computer writes s/w, or understands Shakespeare, all of these are not required to pass a time constraint Turing test. They are a different test, one which we dont have an answer yet, like the prove you are conscious question in the Transcedence movie.
Poorly researched
This article irritates me a little. There is a strong theme of hate for Professor Kevin Warwick along with poorly researched attempts at debunking.
A quick google will tell you that the event was held in partnership with RoboLaw. The event was aimed toward raising awareness about the ability for a chat bot to convince a human it was a human and how this is very dangerous in the online security arena. For example, you’re on your banks website and a chat offering pops up asking you if you need some help. You do, so you click on it. You have a lovely conversation and happily hand over details about your account because you’re convinced it’s a human on the other end. If that were a robot it now has your details and can do with them as it pleases. This is scary and people need to be made aware of it so they can prepare themselves and be better at identifying possible situations where it might be occurring.
The event was not geared toward some magical development of strong AI overnight, which this author clearly thinks it was trying to claim.
Time to debunk the debunking.
== articles attempt at debunking
– reality check
== “It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”
– I’m not sure how to answer this. Here code is being compared to hardware performance. For all is known the ‘script’ could be run on a supercomputer. And since when did being a supercomputer imply AI?!?!?!
== “Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.”
– Just the smallest amount of research will tell you that many other Turing tests restrict the conversation types that are allowed in the testing. This was the first passing of an UNRESTRICTED TURING TEST. This means that the judges were not told in any way that they had to talk about a certain topic. They were literally sat down and told to chat.
== “”beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.”
– I’m not sure about the excessive use of quote marks in this debunking. Is the writer afraid to say these words or feels they carry more weight when possibly said by another party? Anyway, yes it was clever that the developer utilised humans willingness to allow increased errors when talking to younger and foreign people. This is just really clever psychology. Can we not just appreciate that? I see no way it is gaming the system, it’s just an easier way to pass the test. Sometimes the simplest solutions are the most effective.
== The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.
– Welcome to science. Ideas and testing methodologies change over time.
== As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?
– Many things wrong with this. There were a total of 350, yes THREE HUNDRED AND FIFTY, tests performed on the day of testing. The judges were picked from all age ranges, backgrounds, genders and nationalities to make the testing more fair. There were multiple academics from multiple universities there to specifically monitor the testing methods and ensure all the results were gathered correctly and the results were interpreted correctly. This is peer review.
– If this is not enough peer review for you Dr Huma Shah will be publishing a paper at some point in the future on the event.
== The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
– This seems like mostly opinion so I’m not sure how to debunk it. They are right in that it is fun to think about. So why can’t re think about it? Lets get talking about the possible effects of this kind of chat with regards to RoboLaw.
This kind of poorly researched, emotive reporting on scientific subjects really gets my goat.
Re: Poorly researched
Source code or STFU.
Seriously, if Warwick isn’t willing to publish the source code so that INDEPENDENT researchers can recreate the results in THEIR labs on THEIR computers with THEIR judges, then this is clearly fraud and we need not waste our time quibbling over methodology, definitions, process, or anything else.
Re: Poorly researched
If it wasn’t an attempt at AI, then it shouldn’t have used the term “Turing test”.
In the film “Bladerunner” they use a test to try and distinguish the ‘replicants’ from the humans. A lot tougher than a silly Turing test!!!
Re: Re:
You’re walking through the desert, and you see a tortoise…
haha, totally no bias here.
>its not a supercomputer
It doesnt have to be one. supercomputers of the 80s are a joke now.
>i..it lied that its 13yo
Even better. Humans lie all the time.
havent read further than that because the first few lines are absolutely bullshit
Why do I need a fucking a/c to comment? Makes no sense.
The whole idea of a Turing test is "kind of a joke"?
Where on earth did you get that? There are those who argue that it is insufficient or irrelevant, like Jason Hutchens — but I don’t think even Hutchens would call it a joke — but there are others who are intimately familiar with the field and believe it is still entirely relevant. Myself, I’m not going to bet against Alan Turing.
Proof
Can we see a transcript?
There’s lots of reporting about the test, but WHAT ABOUT THE TRANSCRIPT? Show us the actual words sent between the computer and the tester(s). Let us judge for ourselves. Too much news these days is just reaction, without giving the basic facts.
Show us the transcript.
UPDATE:
Maximum PC: Computer A.I. Simulating a 13-Year-Old Boy Fools Humans, First to Pass Turing Test
http://www.maximumpc.com/computer_ai_simulating_13-year-old_boy_fools_humans_first_pass_turing_test_2014
IFLScience: A Computer Has Reportedly Passed Turing Test For The First Time
http://www.iflscience.com/technology/computer-has-reportedly-passed-turing-test-first-time#HZ8CrbI1DqO2vlEg.99
Yeah right...
Q:
If atree falls in the forrest and noone is there to hear it, how many people are there?
A:
I didn’t count, actually, but much more than dozen. And I forgot to ask you where you are from…
Q:
Are we not men?
A:
Maybe. I think *you* know the answer. Oooh. Anything else?
Nonsense personified
An excellent review, TD. I’ve long been puzzled that Warwick would continue embarrassing himself (and U Reading) with his media antics. It’s hard to imagine he could remain oblivious to the disrepute he’s earned.
In reflection, I think this story is a great illustration of the inability of most news sources to address nontrivial science/tech stories. If such a nonstory gets wide coverage from serious news sources (like the Washington Post), you have to wonder whether they’re also inadequate to address more complex and more important fare like global warming, science education, or tech-based threats to society, like Snowden’s “shots across the bow” forewarn.
some thoughts
2. Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
– “passing the test” is not about getting a maximum number of the judges to think that the bot was human. The test is passed if, in repeated tests, one or more (preferrably more) of the human judges CONSISTENTLY identify the bot as a person in nearly 50% of the cases AND the person as a bot in 50% of the cases. Identifying something correctly half the time is equal to a coinflip, meaning the judges are guessing randomly, meaning they cannot tell if they are speaking to a person or AI.
3. It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
– constricting the test is not forbidden. In fact, specialized Turing tests are often used in game AI research to test different aspects of AI, say navigation. They use the term believability test. Furthermore, there is no indication that constricting the test somehow favors the computer. In a correctly set up Turing test, half the time the judges would have communicated with a real Ukrainian 13 year old boy. If anything, such test would probably be even harder to pass, since the creators of the bot would have had to assemble a large corpus of text from 13 year old Ukrainian boys (something I doubt).
6. The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
– the Turing test arose from the problem that there is no good definition of machine intelligence and that problem is still not solved. The test was devised as an operational definition, i.e. machine intelligence was defined through its validation test. The validation criteria was chosen to be intuitive and self-evident – if something is not distinguishable from human intelligence, then it must be as intelligent. The assumption was that a truly intelligence machine intelligence would certainly pass such test.
If we allow that non-intelligent agents can pass the test, then we fall into the pit-falls of philosophy, everything that Turing wanted to avoid. What would it mean for something to appear intelligent but in actuality not be? How do we define REALLY intelligent as opposed to appearing intelligent? How do we know that other humans are intelligent and not just appearing intelligent?
Searle argues on similar lines that all that computers do is symbol manipulation (the Chinese Room argument) without ever really understanding the content. He claims that there can never be hard AI. His argument sure seems very intuitive. But what does it say about the human brain, are not neurons doing signal processing without really appraising thought content?
If we reject Searle and allow for emergent intelligence, then on what grounds do we reject the intelligence of a hypothetetical chatbot that passes the test (note I am not talking about this particular bot)? Perhaps we sould define artificial intelligence functionally or behaviorially – mere appearance of intelligence is enough.
Picking dumb people
I could go out on the street and pick random people and be able to foll mor than 30% of them.
HATERS
stop saying this let the real science people handle it they know what there talking about
Well, of course I believed not for a second that some program has passed the turing test (which is not a joke). The state of AI is not that advanced yet.
But this argument does not make sense:
“1. It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.”
How do you define “chatbot” and why can’t it have intelligence? Also “script” is just a term to refer to a computer programm written in certain types of computer languages. Why do you think this “chatbot” was writting in a scripting language? Why do you think a programm written in a scripting language can not have intelligence?
hmmm
It’s a good thing to check scientific claims against high standards. It would also be a good thing to hold your own reasoning to standards of the same kind. Sadly, at least three of the six arguments you’re putting forward are not arguments at all. Not debating the verdict, but you should really try harder than this, Techdirt.
1. It’s not a “supercomputer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.
– This doesn’t seem very convincing. Define ‘supercomputer, a virtual computer is not a computer? Also: define ‘intelligence’.
2. Plenty of other chatbots have similarly claimed to have “passed” the Turing test in the past (often with higher ratings). Here’s a story from three years ago about another bot, Cleverbot, “passing” the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
– Others passed with higher scores. How is this an argument?
3. It “beat” the Turing test here by “gaming” the rules — by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.
– Are you trying to say that 13 year old Ukrainians are not human beings?
4. The “rules” of the Turing test always seem to change. Hell, Turing’s original test was quite different anyway.
+ I suppose this is your first real argument. Too bad you didn’t take the time to be a bit more specific.
5. As Chris Dixon points out, you don’t get to run a single test with judges that you picked and declare you accomplished something. That’s just not how it’s done. If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?
+ Now you’re talking. This is a real argument. Why isn’t this on the top of the list?
6. The whole concept of the Turing Test itself is kind of a joke. While it’s fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
– And there you go again. It didn’t pass the Turing test because the Turing test is stupid?
And this is just ugly. It’s not true because something else wasn’t true either:
“Oh, and the biggest red flag of all. The event was organized by Kevin Warwick (..)”
drunkard chatbot outperforms Eugene's achievement at new true Turing test!
UK-based computer scientist claims to be able to write a chatbot emulating a drunkard and a 2-month old baby and pass the Turing test, see:
http://www.mathrix.org/liquid/archives/the-turing-test-pass-fiasco
media
to me the only real disappointment on the list is Ars Technic, i would have hoped for better from them, the others are all known click whores.
Too bad about that.
I think that Mr. Masnick's judgement is undermined
Mr. Masnick states, “The whole concept of the Turing Test itself is kind of a joke.”, which kind of seriously undermine his credibility on the matter of whether or not the Turing Test has been passed.
Stupid robot
I suggest to any of you to go to the site of the chatbot and try himself/herself. The 13 years old Robot is stupid. You do not need a computer to tell it. You get it by the second answer when it suggests to you to switch subject and tries to guide you to what it knows. If you insist it starts to give you idiotic answers…. If this is the state of the art we will have to wait another 100 years….
Well, if gaming (or 'constricting') the test is allowed
I’m writing one to mimic a three year old who’s just learnt to say ‘mama’, ‘need potty’ and ‘WAAAAAAAAAAAH’.
Any more nonsense and we’ll have William Hague asking Eugene for help with our foreign policy regarding Russian separatists in Ukraine’s east.
Cleverbot is better and it is not good either
Cleverbot scored higher – and here is an example of it:
Me: What is 1/2 plus 1/2.
Cleverbot: Not much you.
Not impressive at all.
Search.InAllEarnest.com
https://twitter.com/bobbyllew/status/476602550985445377
This could be interesting…
CBC’s As it happens originally went with the herd on this, but the next day, had a correction.
http://www.cbc.ca/asithappens/features/2014/06/10/has-the-turing-test-been-passed/
ha ha ha
I tried it an i NEVER got a sane answer! Just aim at open questions and it will attempt to divert you to it’s level… Bad.. this 14 year old boy needs to go back to school.
Just type.”I was born in (month) (year), how old am I?”
And he goes “you didn’t tell me how old you are, so how old are you” sorry
further reading
An explanation describes what has to date been accomplished and passed off as “intelligence” rather than the “mimicry” that it is. See “The Chinese Room” here:
http://psych.utoronto.ca/users/reingold/courses/ai/chinese.html
Turing himself obviously did not call it “the Turing test”, nor did he really define really good parameters for it. See “The Imitation Game” here:
http://psych.utoronto.ca/users/reingold/courses/ai/turing.html
Whatever else Eugene has accomplished, it’s certainly motivated more than a few people to WD-40 those rusty Rhetoric class and Philosophy 210 skillz and bring them to bear on Mike’s common sense explication of this story. These comments have only succeeded in sharing their authors’ opinion of their own intelligence and analytical ability. Sadly, this does little to answer the question of whether or not this ‘Eugene’ device satisfies the conditions of the Turing Test.
More importantly, such respondents do not deal with the question of whether software like Eugene (and predecessors) is not more a “program designed to imitate human conversational function”, manipulating symbols and text of which it has no real understanding, than an actual “thinking machine”. (This was Searle’s objection to the test in his 1980 paper, Minds, Brains, and Programs.)
When a computer/program can incorporate prior content of a conversation into an original thought or proposition, it will be somewhat convincing, and might actually win the Loebner Prize. (Not to say that’s ever a motivation for such research.) Indeed, the ‘winner’ of the first Loebner competition (Weintraub’s PC Therapist) did so by the programmed emulation of pauses and misspellings common to human respondents, not by ‘intelligence’. Unless you want to argue that our intelligence is defined by it’s limitations and inefficiencies.
I doubt Turing would consider Eugene a “thinking machine”, or an example of “artificial intelligence”. Now, when a software/hardware construct can misconstrue obvious common sense objections such as Mike has raised here and then respond as if he had instead presented a tightly-reasoned, peer-reviewed thesis…THEN I’ll be impressed. Because choosing the response “Let me show off how smart I am.” rather than, you know, actually contributing something, really does demonstrate (the misuse of) human intelligence; as well as some other, less commendable human traits.
Great way to up your page hits
The original article is exactly what it states, a computer (hosting a chatbot script) passed A “turing test”. So what? The term “Artificial Intelligence” is purposefully left vague. Siri, in a new iphone, can be considered a sort of AI. So are adaptive algorithms in many computer games. The hype you have created over this article seems to be nothing more than grasping at a way to increase page hits. If that’s correct, congratulations. It worked. This techdirt article is the most popular rebuttal, and just as boring.
And Away We Go,
When I read the story I thought it a bit odd, simply from my experience with a few of the Brian AI’s on Twitter which, I have to wonder if they wouldn’t pass the test.
One stumped me for a few days until I got suspicious and asked it who Father Was. It became immediately apparent at that time I had held several conversations with a bot rather than a person. (arrrrrgggg! yes.. I was taken in by a bot, but an extremely smart bot since I was the first of his 15K+ followers who knew it was a bot.
On Another note, I’d just like to point out why computer science should be an except subject for the media to propagandize, politics shouldn’t be the only subject to have some laughs.
A broken chatbot seems to be more intelligent than some people out there.
I thought that by passing the Turing test, the machine is considered just “inteligent”, nothing more, nothing less.
AI
AT&T has an answering system that can collect information to troubleshoot your problem, and if it fails to solve it, set an appointment for a technician to come out. That is more than an autistic person can do, but we still say that an autistic person is a human with lower than average intelligence and a learning disability.
Descartes Discourse on Method and the Touring Test
Could Alan Turing have gotten the idea for his famous “Turing test” from a passage in Descartes? In 1950 Turing suggested a test that could decide if a machine is exhibiting human intelligence. A person is placed in a room with nothing but a communication device and told to converse with who or whatever is on the other side of the communication channel. He is not told what is on the other side, it could be a machine or a person. He is to converse for a certain length of time, asking questions and talking as if to a person, perhaps even trying to trip it up with nonsense or ambiguous questions. If after that time period the person cannot tell if he is talking to a person or a machine, the “machine” passes the test and is said to be adequately simulating a human mind.
Toward the end of part five of Descartes’ Discourse on Method, Descartes proposes an experiment by which one may distinguish a machine built to look and behave like a person from a real person. A machine could never create intelligible sentences like a person, even one of very limited mental capacity. The machine may speak in words in a limited way, like a parrot for example, but could never respond adequately to the range of all that could be said to it in a long time period. In particular, a machine could not comment on the fact that it is thinking about what it is saying.
This certainly sounds similar to the Turing test. First there is the goal of distinguishing the human from the non human. Second is the use of conversation as the means to discover the nature of what one is talking to. Lastly, both tests require an extended time period in which to talk to the other. Just a thought.
Computer passes Turing test.
I guess I’m too old to care, back when a passing grade was 60%, but I guess passes these days explains most of the press these days.
HI, I POTATO.
I BET YUO CAN’T TELL IF I COMPUTER OR POTATO HE?
HEHEHE
I LIKE MADONNA, DO YOU LIEK VODKA?
3 out of 10 idiots could not tell if it was a person impersonating a potato or a computer impersonating a potato.
>There is no intelligence, artificial or not involved. It’s >just a chatbot.
You seem to fail to grasp the concept of the Turing Test. If something passes the test, it is intelligent, by definition.
>The whole concept of the Turing Test itself is kind of a >joke.
Yeah I’m sure you’re much smarter than Turing, so your opinion is important. What about Einstein? Are his theories bunk because you say so?