First Test Of Computer Jeopardy Player Goes Well; Watson Beats Mere Humans

from the show-off dept

Last month, we wrote that the IBM computing project Watson was ready to take on real Jeopardy contestants in February of this year (just a few years after the project first came to life). While the big test isn't until Valentine's Day, apparently they had a dry run, and things are looking pretty good for Watson, who beat Jeopardy champions Ken Jennings and Brad Rutter. It wasn't a complete domination. Apparently Watson just narrowly edged out Jennings, though Rutter couldn't keep up. Of course, between now and the real test, Watson can be tweaked. Jennings' and Rutters' brains are pretty much set, and I'd imagine that their ability to cram more useless trivia in their brains is outmatched by Watson.


Reader Comments (rss)

(Flattened / Threaded)

  1.  
    identicon
    Bonzo, Jan 14th, 2011 @ 3:25am

    It's surprising how far AI has come. Not sure how "true" this AI is though, as it's specially configured for the jeopardy contest.

    Still, steps in the right direction. I wonder how long it will be before fully fledged AI computers will be interacting with us on a regular basis!

     

    reply to this | link to this | view in thread ]

  2.  
    icon
    Richard (profile), Jan 14th, 2011 @ 4:06am

    From My Introductory AI Lecture this morning

    Before project:

    "We will construct a program/robot that will perform task X (e.g. play Chess). The resulting program/robot will thus have artificial intelligence - since humans use intelligence to perform task X."

    After project

    Scenario 1

    "We failed"

    Scenario 2

    "We succeeded - however on closer examination it turns out that the method our machine uses is really some form of brute force algorithm that can't really be called intelligent. It seems that you don't really need intelligence to perform task X after all."

     

    reply to this | link to this | view in thread ]

  3.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 4:23am

    When they started losing to Jennings, they probably just tweaked the questions. "And now for another question about Justin Beiber..."

     

    reply to this | link to this | view in thread ]

  4.  
    identicon
    John Doe, Jan 14th, 2011 @ 4:38am

    You seem excited by this

    Don't you know once SkyNet gets started, it won't be stopped? We will be eradicated. Or worse, turned into copper top batteries. ;)

     

    reply to this | link to this | view in thread ]

  5.  
    identicon
    bob563, Jan 14th, 2011 @ 5:20am

    Re: You seem excited by this

    You will now be sued since "copper top battery" is owned by Duracell.

     

    reply to this | link to this | view in thread ]

  6.  
    icon
    Sean T Henry (profile), Jan 14th, 2011 @ 5:48am

    Re:

    I do not really consider this being a big thing. This is like saying a 6th grader beat Ken Jennings using only his skills to type the question in Google.

     

    reply to this | link to this | view in thread ]

  7.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 5:52am

    This is no real contest I'm afraid. This machine is so different from Big Blue that competed in chess. This is simply a search and seizure operation, you ask a question and it searches it's enormous database and finds the answer. For all we know it may just Google it to get the answer. The computer has an unfair advantage unless you give the humans the same access to the data.

     

    reply to this | link to this | view in thread ]

  8.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 5:57am

    Re:

    if you even bothered to read the article, you'd notice that it says it doesn't have access to the internet

     

    reply to this | link to this | view in thread ]

  9.  
    icon
    antimatter3009 (profile), Jan 14th, 2011 @ 6:17am

    Re: Re:

    Jeopardy questions are not straightforward. What's impressive here isn't finding the answer, it's figuring out the question. Natural language processing is very difficult, and Jeopardy provides a particularly difficult version of it, too, with all the puns and such.

     

    reply to this | link to this | view in thread ]

  10.  
    identicon
    Ryan Diederich, Jan 14th, 2011 @ 6:53am

    Much more complex...

    The programming here is much more complex than many realize. This thing has to be able to define the categories, and then figure out what the definition of the category means to each question.

    For example, a question might read: 1812

    The AI would have to remember that the category was "wars in history" and then apply that to its database search.

    Of course that one was simple, there are many more complex ones that I cant think of.

    I do remember one category that was, in other words, "Poets with the same letters for their first and last name"

    Now, unless you were expecting this category, how on earth could you program something to realize what that means.

     

    reply to this | link to this | view in thread ]

  11.  
    identicon
    J\, Jan 14th, 2011 @ 7:41am

    Re: Much more complex...

     

    reply to this | link to this | view in thread ]

  12.  
    identicon
    Jason, Jan 14th, 2011 @ 7:44am

    More difficult than that would be the intuition type categories like Rhyme Time and Before and After. In these cases you have to get the idea of how the answers work before you can start answering correctly.

    What I'm wondering: Are they trying to anticipate common categories and programming this type of information into the computer or simply relying on some kind of intuition algorithm (?).

     

    reply to this | link to this | view in thread ]

  13.  
    icon
    Chronno S. Trigger (profile), Jan 14th, 2011 @ 7:54am

    Re: Re:

    A 6th grader can find the question with a Google search, but that 6th grader has the intelligence to take that answer and turn it into a keyword search, taking into account the word play, and sifting threw the many, many responses for the question that fits.

    Computers aren't good at figuring out what information can be ignored and what words are being used in ways that aren't in the dictionary.

     

    reply to this | link to this | view in thread ]

  14.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 8:00am

    Re:

    It's really not that simple. I can see how it seems that way at first, but in fact this was a massive AI challenge. It's not having the knowledge that is difficult (and no, it has no internet access) - the challenge is parsing the wording of the Jeopardy questions to figure out what the answer is. Remember: computers don't understand English. They just fake it, and usually not very well.

    There are lots of articles out there explaining why Watson was a very real AI challenge - and no they are not all just articles by IBM. You should definitely look more into it because it's actually pretty fascinating and it truly is a major breakthrough, if not in genuine "artificial intelligence" then at least in the natural language processing (which is one of the core building blocks of artificial intelligence)

     

    reply to this | link to this | view in thread ]

  15.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 8:02am

    Re: Re:

    A 6th-grader's brain comes pre-installed with the near-magical capability of abstraction. THAT'S what the challenge was here. Perfect recall is no huge feat - which is why even people like Jennings are only really impressive in a very limited way. But a computer learning to abstract and recognize complex patterns in order to answer natural english questions - that is tough

     

    reply to this | link to this | view in thread ]

  16.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 8:19am

    Re:

    My thoughts exactly. They asked the kinds of questions that computers are good at and worded them in a way that would help the computer answer correctly. They probably also chose questions that the computer was designed to be able to easily answer ahead of time.

     

    reply to this | link to this | view in thread ]

  17.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 8:22am

    Re: Re:

    (and they installed preprogrammed answers ahead of time designed to answer those pre-planned questions. So it's kinda like allowing someone to study a test bank ahead of time and then testing how well that person knows the answers to the questions in the test bank. Very little conceptualization required).

     

    reply to this | link to this | view in thread ]

  18.  
    icon
    Esahc (profile), Jan 14th, 2011 @ 8:57am

    I'll take 'destroy all humans' for for $500 Alex.

     

    reply to this | link to this | view in thread ]

  19.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 9:06am

    Re: Re:

    If I chose the questions, or set of questions to be asked, ahead of time then even I can create a computer program that can answer each question flawlessly and beat almost anyone who is being exposed to those questions for the first time. That's hardly impressive. I will just write a test bank of questions, program them into the computer, program the answers into the computer, and ask those pre - programmed questions.

    Were the people who created the program independent of those who chose the questions? Were those who chose the questions given any guidelines as to what kinds of questions should be asked, what kinds of questions the computer has answers to, and how the questions should be worded? This test run was done at IBM's headquarters and so it was done in a controlled environment. Practically any beginner programmer can create an excellent Jeopardy contender in a controlled environment if the set of questions and answers were predetermined. We must know how prepared the computer was to answer the set of questions that were being asked ahead of time in opposed to how prepared it was to answer any random set of questions.

     

    reply to this | link to this | view in thread ]

  20.  
    icon
    teka (profile), Jan 14th, 2011 @ 9:44am

    Re: Re: Re:

    do at least the tiniest, most minimal fact-checking before you spout off that they were somehow cheating or making things easy for the Watson system.

    otherwise you can come out sounding petulant and foolish.

     

    reply to this | link to this | view in thread ]

  21.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 10:35am

    Re: Re: Re:

    You mix some good concerns in with concerns such as "maybe they cheated entirely". Why don't you give them some credit and analyze it intelligently?

     

    reply to this | link to this | view in thread ]

  22.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 10:35am

    Re: Re: Re:

    What? Are you really accusing the whole shebang of being completely rigged?

     

    reply to this | link to this | view in thread ]

  23.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 10:38am

    Re: Re: Re:

    Where does this whole "pre-chosen questions" concept come from? You are just making that up.

    That's not what happened. And yes, the people creating the program ARE independent. Maybe you didn't notice but this is an IBM computer competing at Jeopardy - IBM does not own Jeopardy and Jeopardy has no interest in helping them win. Besides, both sides have made it clear that the challenge is not altered in any way from a typical Jeopardy match - so unless you are accusing everyone involved of lying to our faces, I have no idea what you are talking about.

    This is a genuine challenge, but you for some reason insist on believing it's a massive cheat. What has given you this idea?

     

    reply to this | link to this | view in thread ]

  24.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 10:48am

    Re: Re: Re:

    Watson's been around for several years now. It wasn't made just so it could answer Jeopardy questions.

     

    reply to this | link to this | view in thread ]

  25.  
    icon
    Chris Rhodes (profile), Jan 14th, 2011 @ 10:50am

    Re: Re: Re: Re:

    Well, they have exactly zero evidence of any cheating having occurred, except for the idea they just manifested in their own minds, so clearly the unethical nature of the developers is a solid fact.

    Clearly.

     

    reply to this | link to this | view in thread ]

  26.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 11:16am

    Re: Re: Re: Re:

    I'm not saying the above is what happened, the above is just an oversimplification of how a computer can beat a person.

    My point is that if the computer has been asked these questions before and the answers that the computers algorithms will provide are known in advanced (and the computer has been tweaked to answer these questions correctly), and the people contending against the computer have never been asked these questions, then it's not as surprising that the computer won. Sure, that still maybe a slight oversimplification, but the point is that the computer will naturally be more familiar with and tuned to answer the specific types of questions that its programmers will tend to ask along with being able to understand the specific wording that the programmers will tend to use (and respond with the expected wording).

     

    reply to this | link to this | view in thread ]

  27.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 3:45pm

    Re:

    Yah! bug you neglect the concept behind it, you take for granted the human ability to do so, is not that simple to make a computer do it, because we don't understand how we work.

    If you doubt please try to make a search engine that understands the meaning of words and can keep track of the many many double meaning things have.

     

    reply to this | link to this | view in thread ]

  28.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 3:52pm

    Re: Re: Re: Re:

    Hmmm...may I interject here?

    Jeopardy was known to fix their games for a long time, this is history not story, so I would be careful not believing they could just lie to our faces they would if they could get away with it LoL

    Other then that I doubt IBM would want that, because that system would be eventually be open to scrutiny and the truth would come out and destroy their good reputation that helps them sell billions in hardware. That is why I don't believe they are cheating here, they don't do that kind of thing because it would cost them a lot of money.

     

    reply to this | link to this | view in thread ]

  29.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 3:59pm

    Re: Re: Re: Re: Re:

    If they did, their AI business that administer things like red lights, cameras for traffic control etc would be in serious jeopardy(pun intended) here.

     

    reply to this | link to this | view in thread ]

  30.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 5:22pm

    Re: Re: Re: Re: Re:

    Hmm - I didn't know that about Jeopardy. But definitely looking at this situation there are no benefits to either side for cheating: Jeopardy would rather have a human win, because the humans sort of represent them in this rivalry - and a computer beating their game would trivialize it in a lot of people's eyes. Meanwhile IBM, as you say, gains nothing by pretending to have made a breakthrough in AI, except perhaps a momentary stock bump before they get found out and torn to shreds.

     

    reply to this | link to this | view in thread ]

  31.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 5:29pm

    Re: Re: Re: Re: Re:

    You are making this all up. IBM has no idea what questions the computer will face - sure it has some concept of the type of question, just like any human practicing to be on Jeopardy. But the questions Watson faces are new and unexpected.

    And the whole concept of a "type" or "nature" of question is exactly the point here. Humans can abstract and recognize the "type" of a bunch of complex and varying things like trivia questions. To a computer, each one is a completely unique string of characters that can only be parsed with strict logical conditions - same goes for every single one of the facts it has in its database of knowledge.

    This is obvious even at basic levels of computing: spellcheckers work well, grammar checkers do not. That's because spelling is a (for the most part) strictly defined set of rules - the computer simply checks each word against its dictionary, seeking an exact match. Grammar, on the other hand, is highly complex with varying degrees of strictness and countless structural options: there are "rules" but they are not nearly as simple or clear, always being full of exceptions and intricate conditions. Now, even though most people cannot explain all the rules of grammar, we instinctively know when a lot of things are right or wrong just based on how it "sounds" to us - and we can't even explain what we mean by that, or how that process works in our head. Computers have no such skill.

     

    reply to this | link to this | view in thread ]

  32.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 5:33pm

    Re: Re: Re: Re: Re:

    Wait, I just figured it out - I think he's thinking of the Jeopardy! video game

     

    reply to this | link to this | view in thread ]

  33.  
    icon
    Marcus Carab (profile), Jan 14th, 2011 @ 5:34pm

    Re: Re: You seem excited by this

    Another companies really needs to market their batteries upside-down and call them Copperbottoms.

     

    reply to this | link to this | view in thread ]

  34.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 6:35pm

    Re: Re: Re: Re: Re: Re:

    "I didn't know that about Jeopardy."

    Many game shows have been known to do that, not just Jeopardy, though that wasn't my point here. My point here, from the article

    "was a test run this morning at IBM Research's headquarters in preparation for a televised weekend challenge"

    From my understanding, this wasn't done by Jeopardy, it was done by IBM. To what extent were the selected questions created by people not involved in the development of the computer hardware, software, and database info and to what extent did those people creating the questions receive guidance as to how to word the questions, how accepted answers should be worded, and what kinds of questions to ask and which ones to steer away from.

     

    reply to this | link to this | view in thread ]

  35.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 7:00pm

    Re: Re: Re: Re: Re: Re: Re:

    (and guidance as to what expected answers to expect so that questions can be biased towards ones with those answers). I'm not saying it's necessarily fixed, just that those developing the system will naturally have a tendency to ask certain kinds of questions, even if they're not conscious aware of it.

     

    reply to this | link to this | view in thread ]

  36.  
    identicon
    Anonymous Coward, Jan 14th, 2011 @ 7:00pm

    Re: Re: Re: Re: Re: Re: Re: Re:

    consciously aware *

     

    reply to this | link to this | view in thread ]

  37.  
    icon
    vivaelamor (profile), Jan 15th, 2011 @ 6:50am

    Re: Re:

    "I do not really consider this being a big thing. This is like saying a 6th grader beat Ken Jennings using only his skills to type the question in Google."

    Another point worth making is that Watson is completely offline. While they feed it presumably more books than any human could read in their lifetime, it cannot just rely on the question having already been answered in an easy digest form on Google.

     

    reply to this | link to this | view in thread ]

  38.  
    icon
    vivaelamor (profile), Jan 15th, 2011 @ 6:56am

    Re: Re: Re: Re: Re: Re:

    'And the whole concept of a "type" or "nature" of question is exactly the point here. Humans can abstract and recognize the "type" of a bunch of complex and varying things like trivia questions. To a computer, each one is a completely unique string of characters that can only be parsed with strict logical conditions - same goes for every single one of the facts it has in its database of knowledge.'

    Two terms worth throwing in here for anyone wanting to learn more about the sort of processes I assume Watson uses: fuzzy logic and neural network.

     

    reply to this | link to this | view in thread ]

  39.  
    icon
    vivaelamor (profile), Jan 15th, 2011 @ 7:04am

    Re: Re: Re: Re: Re: Re: Re:

    "From my understanding, this wasn't done by Jeopardy, it was done by IBM. To what extent were the selected questions created by people not involved in the development of the computer hardware, software, and database info and to what extent did those people creating the questions receive guidance as to how to word the questions, how accepted answers should be worded, and what kinds of questions to ask and which ones to steer away from."

    Why would IBM cheat on the not actual Jeopardy test? It's obvious why they wouldn't cheat: they can test it against the actual competitors and tweak the systems in preparation for the real test. Are you seriously suggesting that they're not confident of their chances in the real test and are using this trial run to make it look like they did well?

     

    reply to this | link to this | view in thread ]

  40.  
    icon
    vivaelamor (profile), Jan 15th, 2011 @ 8:05am

    Re:

    "What I'm wondering: Are they trying to anticipate common categories and programming this type of information into the computer or simply relying on some kind of intuition algorithm (?)."

    I'm guessing that if they needed to then they could a combination. I think it is more likely that such a step would be redundant because such a thing should be easily and more effectively learned by the machine anyway, e.g. by inputting previous actual questions. If every question that had the word 'dig' in the title turns out to be about archeology then the computer will learn that.

     

    reply to this | link to this | view in thread ]

  41.  
    icon
    vivaelamor (profile), Jan 15th, 2011 @ 8:07am

    Re: Re: Re: Re: Re: Re: Re: Re:

    "I'm not saying it's necessarily fixed, just that those developing the system will naturally have a tendency to ask certain kinds of questions, even if they're not conscious aware of it."

    What do you base that on? If their goal is to improve the system then they may well ask it questions that it is less likely to get right, a completely opposite bias to the one you assume.

     

    reply to this | link to this | view in thread ]

  42.  
    icon
    Marcus Carab (profile), Jan 15th, 2011 @ 9:18am

    Re: Re: Re: Re: Re: Re: Re:

    Yes! But then, why learn all these fascinating new things about cognitive processes and artificial intelligence when you can just claim the whole thing is a hoax, right?

     

    reply to this | link to this | view in thread ]

  43.  
    icon
    nasch (profile), Jan 15th, 2011 @ 5:05pm

    Re:

    Anything already invented is just a program, artificial intelligence is always whatever we don't know how to do yet. 20 years ago if you had described a computer that can beat a Jeopardy champion without assistance, that would probably be thought of as AI. Now to be AI it has to, I don't know, be able to both drive a car and cook a meal. And after that's invented, it won't really be AI until it can understand human emotions. And so on. ;-)

     

    reply to this | link to this | view in thread ]

  44.  
    icon
    nasch (profile), Jan 15th, 2011 @ 5:10pm

    Re: Re:

    Remember: computers don't understand English.

    I think you could make a pretty good argument that this one does. Which is what's so awesome about it.

     

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This