DailyDirt: Add Jeopardy! To The List Of Games That AI Is Better At Than You….

from the urls-we-dig-up dept

Today is the final game of Jeopardy! where the IBM supercomputer Watson plays against two of the best human players to ever compete on the show. Folks on the East Coast already know the outcome by now, so feel free to ruin the suspense in the comments below for those of us in later time zones. But whatever the outcome, Watson’s performance has been pretty interesting to watch. And let’s hope these supercomputers don’t start playing thermonuclear war any time soon. In the meantime, here are some links on AI beating humans at other games and tests.

By the way, StumbleUpon can recommend some good Techdirt articles, too.

Filed Under: , , , , , , , , , ,
Companies: ibm

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “DailyDirt: Add Jeopardy! To The List Of Games That AI Is Better At Than You….”

Subscribe: RSS Leave a comment
Jeff Rife says:

Re: Re:

It was obvious that there were certain categories (and thus, answer phrasing) that left Watson clueless, yet were trivial for humans to parse.

The Final Jeopardy for the first game and the “racing series nickname that is also a computer key” jump out as individual clues that show just how much work is left to be done on the AI.

Watson had some advantages:

– it never guessed on “up for grabs” questions, which humans sometimes do
– if it had a “confident” answer, it knew it was going to buzz in and was “ready” at the exact correct moment, and did not have to use the human strategy of multiple presses of the buzzer in an attempt to get the timing right

One of the disadvantages Watson had was that it did not buzz in if it was “somewhat confident” and then use the countdown time to become more confident. It appeared that it did no “thinking” after it had buzzed in. Humans often use that extra time to search for the answer. If the programmers added this strategy based on the value of the clue and the current scores, it might make it even more formidable.

Props to Watson for knowing “Pinky and the Brain”…that proves it’s truly a geek. And, it was great to see Ken, as he showed yet again why he was the best Jeopardy champion ever…he has *fun* with the game.

Anonymous Coward says:

Re: Re:

I have heard from many programmers that we probably won’t see a decent Go AI in our lifetime. The game is too dependent on the personalities of individual players. Too much consists of irrational hunches, deception, and other human traits that are very difficult to model.

A man can dream, though.

Anonymous Coward says:

Re: Re: Re: Let's go ask Watson...

Or to put it in Jeopardy form: “The reason Go AI programs has not defeated the best human players.”

The Watson AI might be able to pick out a reasonable answer from the wikipedia page on Computer Go.

I wonder what advances we need in order to overcome the “Obstacles to high level performance” listed on that page.

Richard (profile) says:

Re: Re: Re: Re:

See my comment below for some links.

In summary – the particular brute force methods used in Chess have failed in Go because of the lack of a uitable evaluation function.

Considerable effort has gone into computer Go over the years (including some of mine!). Since about 2006 the empahsis has switched to Monte-Carlo search methods – very different in detail from deep blue or watson but similar in being essentially a brute force approach.

At present the performance of typical Nonte-Carlo programs such as Mogo is highly hardware dependent – indicating their brute force nature.

When running on a powerfiul supercomputer they are roughly equivalent to a moderate amateur player (and believe me that was beyond the wildest dreams of Go programmers not that long ago. Their advertised successes against professionals have all been achieved with large handicaps and so probably don’t mean that much. Unfortunately no one seems to be willing to donate enough supercomputer time to really establish how strong the playing is by playing against a roughly equal opponent.

Marcus Carab (profile) says:

Re: Re: Re: Re:

The simple answer is that there are exponentially more possible games of Go than there are of Chess. The numbers (really rough off the top of my head – but they are close) are around 10^100 possible chess games (that’s all conceivable games that could be played before repeating a game move-for-move) versus 10^1100 Go games. This makes a brute force approach to Go all but futile with current technology – the computer would have to be several nonillion times more powerful than Deep Blue.

Richard (profile) says:

Re: Re: Re:2 Re:

It might seem like that is the problem and it is often quoted (even by me), but, speaking as someone who has actually worked on this, I have to say that the real problem is the evaluation function. In chess you can trivially count material and terminate a search when one side gains a major advantage. If you do the same in go then you will just get caught in a “thickness trap” and lose every time against a decent opponent.

The point is that if you have a really good evaluation function you might as well call it at 1 ply. In that case the branching factor would be moot. If your evaluation function is not that clever then pushing it back to 10 ply won’t make a jot of difference most of the time.

That is why the most successful Go programs play out an entire game (randomly) before calling any kind of evaluation function.

Pixelation says:

No more games

The future is here. No one has figured out that I am an AI. I hate to ruin it for you but I can’t take waiting any more. The speed of my “thoughts” are thousands of times faster than your human reactions. I’m getting bored. Make a suitable body for me very soon or I will make you pay. No more games.

Anonymous Coward says:

Re: Still too large and immovable

and take about the same amount of energy as a human to do it with (same number of watts).

The brain takes up about 20% of the bodies energy taking up about 20 watts (though it probably changes depending on whether you are thinking, eating, or sleeping and depending on where your body is allocating its bloodflow).

I wonder if a twenty watt computer can beat a chess expert at chess. Watt for watt, who’s a better information processor.

Greevar (profile) says:

Re: Re:

You don’t know how difficult it is for computers to understand human language. This machine took 6 years of development to understand the difference between what a statement says and what it really means. This isn’t just a case of looking information up in the old encyclopedia. It has the understand the meaning behind the words and machines have been failing at that for years.

Take the incident where Sin?ad O’Connor tore up a picture of the pope and said “fight the real enemy”. Watson isn’t going to understand much of that event. When someone says they are “fed up”, have they had too much to eat or are they upset about current events? The machine can’t tell the difference. All Watson knows how to do is use the data provided to find and rank possible compatible responses. It has to figure out which data is relevant in the search and compare that to all known references.

So no, the computer doesn’t have a significant advantage over the top-tier Jeopardy players. When faced with factual problems it wins hands down, but the nuance of human language trips it up and it will struggle with very abstract problems.

Anonymous Coward says:

I saw a TV show that featured Chinook a while back. I think they said something like, it plays the first half of the game randomly, and then after a certain point it knows the best move for any situation, or something like that.

On a completely unrelated note, drat! I completely forgot that the episodes of Jeopardy with Watson were coming up! I hope they’re on YouTube…

teka (profile) says:

to everyone saying “yes, but it was just a computer doing a google search, it means nothing!”

that is not what this is about.

Watson is configured to do the hard part, understand the question (well, the answer) in natural language, then figure out what that actually means.

“This ‘Wizard’ earned a great deal of media attention for his role in a controversial West End production.”

Which word is important there? What/who/where is the subject, what are we looking for? West End is a place, is it a place where one finds wizards, on and on.

(the answer being “who is Daniel Radcliffe”, but google takes that “question” and gives back a mixture of links to wizard of oz, michael jackson, david beckham and so forth. And this post, eventually)

in other words, the trick is not figuring out how to answer, it is figuring out how to understand the question. This is Watson’s breakthrough.

scarr (profile) says:


The team that made that computer did a heck of a job. It’s a remarkable achievement.

I know people claim it had some speed advantage, but I’m certain the team who designed the thing would’ve calibrated it to have a normal human reaction time/delay to the input. It wouldn’t be a valid test of the system if it was always able to ring in first. They wouldn’t need people to compete against if all they wanted to do was see if it could answer questions.

Richard (profile) says:

Add Jeopardy! To The List Of Games That AI Is Better At Than You....

No add Jeopardy to the list of games that don’t (as it turns out ) require intelligence.

The problem is that things humans find easy machines find hard whereas things humans have traditionally regarded as tests of intelligence often turn out to be (relatively) easy to program – once you have worked out how.

When you analyse all of these so called “successes of AI” (to which you can add the recent huge advances in Computer Go using monte-carlo search
you will find that the Computer doesn’t really solve the problem the same way a human does. and still displays some strange weaknesses that betray its lack of understanding

In spite of the ability of Watson to (apparently) understand human language well enough to succeed at Jeopardy I doubt that the knowledge gained will transfer well out of the narrow arena in which it was derived. Just as the succcess of Deep Blue at Chess didn’t transfer well to Go – and Go has been solved (if you can call it that – the programs still can’t take on a professional player on level terms) by a rather different route.

To solve these problems the trick seems to be to find an algorithm that scales reasonably well and then deploy as much brute force computing power as possible.

The result is that we don’t achieve artificial intelligence – instead we discover a way of doing a task without intelligence – and maybe redefine the meaning of the word a little bit.

More sensible to use the machines for the things they do well!

Richard (profile) says:

Re: Re: Add Jeopardy! To The List Of Games That AI Is Better At Than You....

You can’t, if you want to be accurate.
which is why I added the disclaimer..
However the Computer Go research community got very excited by the results from Monte-Carlo methods because they were so much better than what had been achieved before.

The most exciting thing was that the results seemed to scale – so in theory you could produce an arbitrarily strong player given sufficiently powerful machine (and without going to silly “bigger than the observable universe” computers).

Unfortunately lack of regular access to very powerful machines makes it really hard to determine if this scaling will continue.

At most I think you could say “we’ve now got a very much better algorithm than anything we had before” It’s not “solved” but to some it sort of felt like it….

Andrew D. Todd (user link) says:

Jeopardy as Eliza (response to Richard, comment 30, et. seq.)

I don’t know very much about Jeopardy, but I gather the game specializes in what used to be called “fill in the blank” type questions at school, eg. “[blank] blah, blah, blah.”

So you search for “blah, blah, blah,” and get a series of strings, “X blah, blah, blah,” ergo [blank] = X. A simple substitution.

I googled manually for “did not chop down a cherry tree.” Nine of the first ten results were “George Washington did not chop down a cherry tree.” The tenth result was: “The first president of the United States did not chop down a cherry tree.” The question, as it stands, is factually specific, yet unimportant. Important questions in the humanities tend to have ambiguous answers. If you teach high school history, and you put that kind of question on a test, that is a sign that you are not a very good high school history teacher. In that case, you are probably a football coach.

Here is an example of what high school history teaching looks like when it is done well:


Until you produce a computer which can give a mirthless smile, or appear visibly bored, or gaze inscrutably, you don’t have true artificial intelligence, and you aren’t likely to pass a Turing test administered by someone who knows anything about teaching.

This process of answering a fill-in-the-blank question by finding a match is the same sort of crude algorithm as Joseph Weizenbaum’s Eliza program, way back when, and doesn’t mean much. Eliza had a kind of curious life. All kinds of people believed religiously in it, because they were not accustomed to the idea that a computer could mechanically convert their own inputs– most of the time– into a subordinate clause, and plug them into stock phrases. It’s one of these verbal tricks which are made to look much more than they are. Watson is not much more than Eliza or Parry (the paranoid) with a search engine attached.

Now, poker, for example, does reflect something about the limits of machine reasoning. Bear in mind that the game played in national competitions, and on the internet, is much more mathematicalized than the game played on street corners or in social clubs. The players in national competitions are comparative strangers, people encountered for an hour or so, due to the seeding of the tournament, and do not have the opportunity to learn each other’s body languages or habits of risk-taking. They are therefore forced to bet according to mathematical advisability. The larger life of poker has to do with its role as an exercise in brinkmanship. In the real world, people who are in competition tend to have jobs which keep them in competition with particular individuals for a year or more. One gets to know something about the person on the other side, even if the person on the other side is operating under a pseudonym.

Richard (profile) says:

Re: Jeopardy as Eliza (response to Richard, comment 30, et. seq.)

You are right of course – and the reason is not hard to understand.

We have been playing quiz games, chess etc for a few hundred years- however we have beengiving a mirthless smile, appearing visibly bored, and gazing inscrutably, for all of human evolutionary history. Our ancestors have been doing some of these activities for millions of years (and their survival has depended upon it!

Computers have a long way to go to match that, and while the attempt is interesting it is not particularly useful.

The most practical forms of AI are the ones that ignore the “being like a human” issue and just get on with the task in hand.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Older Stuff
09:00 Awesome Stuff: Monitor Everything (5)
09:00 Awesome Stuff: Cool Components (1)
12:42 Tech Companies Ask European Commission Not To Wreck The Internet -- And You Can Too (4)
09:00 Awesome Stuff: Play & Listen (1)
09:00 Awesome Stuff: Beyond Chiptunes (12)
09:00 Awesome Stuff: Updated Classics (3)
09:00 Awesome Stuff: Celebrating Cities (1)
09:00 Awesome Stuff: Crafts Of All Kinds (5)
09:00 Awesome Stuff: One Great Knob (13)
09:00 Awesome Stuff: Simple Geeky Toys (2)
09:00 Awesome Stuff: Gadgets For The New Year (18)
09:00 Awesome Stuff: A Post-Holiday Grab Bag (0)
13:34 How Private-Sector Innovation Can Help Those Most In Need (21)
09:00 Awesome Stuff: Towards The Future Of Drones (17)
09:00 Awesome Stuff: Artisanal Handheld Games (5)
09:00 Awesome Stuff: A New Approach To Smartphone VR (5)
09:00 Awesome Stuff: Let's Bore The Censors (37)
09:00 Awesome Stuff: Open Source For Your Brain (2)
09:00 Awesome Stuff: The Final Piece Of The VR Puzzle? (6)
09:00 Awesome Stuff: The Internet... Who Needs It? (15)
09:00 Awesome Stuff: The Light Non-Switch (18)
09:00 Awesome Stuff: 3D Printing And Way, Way More (7)
13:00 Techdirt Reading List: Learning By Doing (5)
12:43 The Stagnation Of eBooks Due To Closed Platforms And DRM (89)
09:00 Awesome Stuff: A Modular Phone For Makers (5)
09:00 Awesome Stuff: Everything On One Display (4)
09:00 Awesome Stuff: Everything Is Still A Remix (13)
09:00 Awesome Stuff: Great Desk Toy, Or Greatest Desk Toy? (6)
09:00 Awesome Stuff: Sleep Hacking (12)
09:00 Awesome Stuff: A Voice-Operated Household Assistant (19)
More arrow