Share/E-mail This Story

Email This

This feature is only available to registered users. Register or sign in to use it.

Why Netflix Never Implemented The Algorithm That Won The Netflix $1 Million Challenge

from the times-change dept

You probably recall all the excitement that went around when a group finally won the big Netflix $1 million prize in 2009, improving Netflix's recommendation algorithm by 10%. But what you might not know, is that Netflix never implemented that solution itself. Netflix recently put up a blog post discussing some of the details of its recommendation system, which (as an aside) explains why the winning entry never was used. First, they note that they did make use of an earlier bit of code that came out of the contest:
A year into the competition, the Korbell team won the first Progress Prize with an 8.43% improvement. They reported more than 2000 hours of work in order to come up with the final combination of 107 algorithms that gave them this prize. And, they gave us the source code. We looked at the two underlying algorithms with the best performance in the ensemble: Matrix Factorization (which the community generally called SVD, Singular Value Decomposition) and Restricted Boltzmann Machines (RBM). SVD by itself provided a 0.8914 RMSE (root mean squared error), while RBM alone provided a competitive but slightly worse 0.8990 RMSE. A linear blend of these two reduced the error to 0.88. To put these algorithms to use, we had to work to overcome some limitations, for instance that they were built to handle 100 million ratings, instead of the more than 5 billion that we have, and that they were not built to adapt as members added more ratings. But once we overcame those challenges, we put the two algorithms into production, where they are still used as part of our recommendation engine.
Neat. But the winning prize? Eh... just not worth it:
We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.
It wasn't just that the improvement was marginal, but that Netflix's business had shifted and the way customers used its product, and the kinds of recommendations the company had done, had shifted too. Suddenly, the prize winning solution just wasn't that useful -- in part because many people were streaming videos rather than renting DVDs -- and it turns out that the recommendation for streaming videos is different than for rental viewing a few days later.
One of the reasons our focus in the recommendation algorithms has changed is because Netflix as a whole has changed dramatically in the last few years. Netflix launched an instant streaming service in 2007, one year after the Netflix Prize began. Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.
The viewing data obviously makes a huge difference, but I also find it interesting that there's a clear distinction in the kinds of recommendations people that work if people are going to "watch now" vs. "watch in the future." I think this is an issue that Netflix probably has faced on the DVD side for years: when people rent a movie that won't arrive for a few days, they're making a bet on what they want at some future point. And, people tend to have a more... optimistic viewpoint of their future selves. That is, they may be willing to rent, say, an "artsy" movie that won't show up for a few days, feeling that they'll be in the mood to watch it a few days (weeks?) in the future, knowing they're not in the mood immediately. But when the choice is immediate, they deal with their present selves, and that choice can be quite different. It would be great if Netflix revealed a bit more about those differences, but it is already interesting to see that the shift from delayed gratification to instant gratification clearly makes a difference in the kinds of recommendations that work for people.

Reader Comments

Subscribe: RSS

View by: Time | Thread

  1. identicon
    frosty840, 13 Apr 2012 @ 12:48am

    Sure there's a difference

    It's explained in the quote. The difference is that the rental algorithm has to work because satisfactory rentals depend on the suggestion being correct, so you need a 1:1 recommendation:success ratio.

    The streaming algorithm, on the other hand, only has to achieve, say, a 5:1 recommendation:success ratio because if you get recommended five movies, stream two or three minutes of each, hate four and only watch one, then you've still watched a good movie, and you're probably satisfied.

    It's a transformation from customer satisfaction correlating directly with the recommendation algorithm under a rental business model to a much looser correlation under the streaming model.

    reply to this | link to this | view in thread ]

  2. identicon
    Call me Al, 13 Apr 2012 @ 2:43am

    "That is, they may be willing to rent, say, an "artsy" movie that won't show up for a few days, feeling that they'll be in the mood to watch it a few days (weeks?) in the future, knowing they're not in the mood immediately"

    This is very much the case for me. I use Lovefilm over in the UK (similar to Netflix) and there is a film I keep adding to my queue, they keep sending me and I keep sending back. I really want to watch it but I'm just never in the right mood but I hate holding onto the DVDs long term so send them back in short order.

    Also looking at my queued films vs what I watch on instant there is a noticible difference. I watch a lot of trash on the instant player, films that I would never buy but that I'm happy to try out of curiosity at that moment. The queue though is full of films which are rated well and which I generally look forward to receiving in the post.

    Obviously there is also the consideration that some films which I would watch on instant if available are not there so end up queued.

    reply to this | link to this | view in thread ]

  3. identicon
    Anonymous Coward, 13 Apr 2012 @ 3:03am

    This is called crowdsourcing. Companies do this in order to avoid actually having to - gasp! - pay for an actual employee.

    People work for free under the guise of a contest, and the best work is chosen and the author is given some bullshit prize for placation.

    Designers are the ones who fall for it, given lack of experience, but programmers and computer scientists?

    reply to this | link to this | view in thread ]

  4. identicon
    Anonymous Coward, 13 Apr 2012 @ 3:04am


    Indeed. People are more willing to explore less famous titles on "Try and feel" approach. If I'm told that my decision may take time to change in the future, my decision would be more conservative.

    reply to this | link to this | view in thread ]

  5. identicon
    Anonymous Coward, 13 Apr 2012 @ 3:29am


    Don't worry, if anyone deserves a $1,000,000 bullshit prize, it's you.

    reply to this | link to this | view in thread ]

  6. identicon
    Anonymous Coward, 13 Apr 2012 @ 5:10am


    You don't like puzzles, do you? I don't know about other programmers and computer scientists, but I love working on challenging problems. It beats the reports and applications that I am paid to work on.

    reply to this | link to this | view in thread ]

  7. identicon
    Mike, 13 Apr 2012 @ 5:38am

    Good point on the psychological differences between choosing videos for your future self vs your current self.

    Made me immediately think of Natalie Tran's observations on current self vs past self. If you haven't seen the video it's here:

    One of the most practical KDDCUP competitions - a pity the competition didn't match more closely the engineering requirements.

    reply to this | link to this | view in thread ]

  8. identicon
    pr, 13 Apr 2012 @ 6:40am

    I think they have it nailed.

    They describe my behavior exactly. My DVD list had a bunch of deep, culturally significant things I would like to be able to say I've seen. Sometimes they would sit around for a month before we sent them back unwatched. Now whenever I want ten minutes of thoughtless pleasure I just slap on an episode of "Shaun the Sheep."

    I usually watch them one skit at a time. I hope they don't take partially watched to be evidence that I don't like it, it's just that that sort of comedy is better in small doses.

    reply to this | link to this | view in thread ]

  9. icon
    Benjo (profile), 13 Apr 2012 @ 7:40am


    I'm sure the Engineers that came up with the Machine learning algorithm don't give 2 damns if Netflix used their algorithm, since they got paid.

    What's interesting about this is that their suggestion system for streaming videos still affects the company's value prop. And the whole point of the machine learning algorithm is that based on a sample set of millions of input data, they can extrapolate a rating system that holds true for billions of future cases (at least better than their current system).

    What I'm saying is they have even more data now, and they could probably pay some engineers to build a new machine learning algorithm for streaming videos, with a very good chance of making a much more accurate system. I'd rather see them spend money on that then trying to launch their own cable channel.

    reply to this | link to this | view in thread ]

  10. identicon
    dev, 13 Apr 2012 @ 7:45am

    Canadian selection

    in canada, the issue is more of a lack of quality to select from. netflix is constantly recommend things that are barely three stars. when i joined i hoped netflix would recommend new things to me that i hadn't heard of, instead i get straight to tv knockoffs.

    I wish when i searched for a streaming movie, that it could play the movie trailer.

    netflix is like searching through the discount dvd bin at walmart.

    reply to this | link to this | view in thread ]

  11. icon
    Benjo (profile), 13 Apr 2012 @ 7:56am

    Re: Sure there's a difference

    I agree with you, but I still think accuracy for streaming recommendations can be pretty important. If it takes you five recommendations on average to find one show you like, chances are your faith in the recommendation is lower than say if it only took you one to three (on average). This in turn lowers the value of their recommendation system, which lowers the value of their service.

    reply to this | link to this | view in thread ]

  12. icon
    Benjo (profile), 13 Apr 2012 @ 8:06am

    Re: Re:

    Also, looking back at the privacy concerns for how they conducted these contests, I'm really surprised Netflix didn't just keep the data themselves.

    I believe when you are coming up with a machine learning algorithm, it is considered bad practice to personally look at the data set you are given. You really only need to know the format and size of the data.

    If it was a more collaborative effort, they could have worked with the research teams and basically run the algorithms themselves, giving results back to the teams. This way, the data would never leave Netflix, and there would be no privacy issues.

    reply to this | link to this | view in thread ]

  13. icon
    mattarse (profile), 13 Apr 2012 @ 9:09am

    Re: Re:

    Exactly - most of my colleagues all have a hobby that is in the same field, but not exactly applicable to their jobs. We enjoy the challenge.
    I remember reading about this prize when it was still in competition and at least a few of the challengers didn't seem to think they had a chance to win, but kept working on it making incremental progress.

    reply to this | link to this | view in thread ]

  14. icon
    jupiterkansas (profile), 13 Apr 2012 @ 9:25am

    Re: Canadian selection

    Your recommendations will improve as you rate more movies. That's the secret ingredient of Netflix. Don't like TV knockoffs? Rate them one star and Netflix will stop recommending them.

    I find Netflix's recommendations to be excellent, but then I've rated 3,680 movies.

    reply to this | link to this | view in thread ]

  15. icon
    sunk818 (profile), 13 Apr 2012 @ 11:54am

    Re: Re: Sure there's a difference

    Accuracy is even more important as the instant gratification requirement is that much higher. I think it is part algorithm issue, but it is also part content issue. If your tastes don't match up with their selection, no amount of algorithm will fix lack of content.

    reply to this | link to this | view in thread ]

  16. identicon
    Asset Tracking Software, 13 Apr 2012 @ 12:57pm

    Actually, that makes sense

    In business they always say you base decisions on the future and not on the past.

    My initial response was to say that they spent $1 million getting a better algorithm and they didn't use all of it, what were they thinking?

    But then, if it didn't make sense to use it when they finally got the algorithm then don't use it. It would be stupid in this situation to use it.

    reply to this | link to this | view in thread ]

  17. icon
    heyidiot (profile), 13 Apr 2012 @ 1:18pm

    $1,000,000 is bullshit?

    Thank you for visiting, Mr Buffett.

    reply to this | link to this | view in thread ]

  18. identicon
    Anonymous Coward, 13 Apr 2012 @ 1:45pm

    Re: Canadian selection

    technically possible, and probably implemented soon.

    reply to this | link to this | view in thread ]

  19. identicon
    BlueRaja, 13 Apr 2012 @ 4:08pm

    Garbage in, garbage out

    Garbage in, garbage out. If I have four people in my family all rating movies, they're not getting a correlation between how a person will rate similar movies, they're just getting a bunch of seemly random ratings. Without fixing this, no algorithm could ever do well.

    Netflix could have fixed this years ago about allowing multiple users under one account, but they never cared. That's why the ratings on Netflix have turned to crap, and (that lack of caring) is why Netflix is itself turning to crap.

    reply to this | link to this | view in thread ]

  20. icon
    nadaAS67 (profile), 16 Apr 2012 @ 12:40am

    management essay

    Thanks, to shearing this nice information, I really appreciate your thinking, & i will visit again for getting more updates.

    management essay

    reply to this | link to this | view in thread ]

  21. identicon
    Sephen, 16 Apr 2012 @ 11:37am

    steaming netflix movies cannot be recomended accurately

    They cannot recommend accurately for me because movies I'm interested in are *not* streamed; that's why I gave up on that, and when they did the split I stayed with dvds rather than both dvds and streaming or streaming alone.

    The movies they offer for streaming could more accurately be offered steaming, as in a steaming pile of dog poop. Until they consistently offer movies I'd be interested in, they can keep their steaming content; I don't want it.

    reply to this | link to this | view in thread ]

  22. identicon
    Anonymous Coward, 28 Aug 2012 @ 3:28pm


    Someone, please provide some competition for this douche of a company.

    reply to this | link to this | view in thread ]

  23. identicon
    Anonymous Coward, 6 Mar 2017 @ 9:20am


    reply to this | link to this | view in thread ]

  24. icon
    firebird2110 (profile), 2 Sep 2017 @ 12:33am

    Re: Garbage in, garbage out

    But Netflix HAS multiple users under one account.

    reply to this | link to this | view in thread ]

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: I Invented Email
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Report this ad  |  Hide Techdirt ads
Recent Stories
Report this ad  |  Hide Techdirt ads


Email This

This feature is only available to registered users. Register or sign in to use it.