Why The Press Is Getting The Wrong Message Out Of The 'Nate Silver Walloped The Pundits' Story
from the small-sample-sizes dept
Of course, if you look at what's happened to baseball since "Moneyball" and the success of the first statistical analysis guys, it should be a reminder that statistical prognostication is still about the probabilities -- and not about true predictions. And this is where the "suddenly-in-awe" pundits are still getting confused. They seem to think that Silver or other statistical modelers suddenly have a magic crystal ball with which they can predict the future. But probabilities and predictions are different, and Silver himself would likely admit (and, actually, did admit) that when you're dealing in probabilities, you're still going to be completely wrong some percentage of the time (he can even tell you what percentage of the time!) Even if the probabilities show a 90% likelihood that a certain event will happen, it still means that one time out of 10, you're going to be wrong.
Unfortunately, our brains don't deal that well with probabilities. We don't think in probabilities. Because we're dealing with a (mostly) binary situation, we assume that as soon as the probabilities tilt in our favor, it means that a "win" is somehow assured, and mentally, the probabilities turn into a prediction. It's very, very difficult for our brains not to think that way.
So I'm thrilled to see statistical analysis "win" over the moronic pundit-class who thinks that "storylines" or "momentum" (or, um, the ultimate in believing in anecdotes over data, "my friends see more yard signs" for one candidate) are valid methods for prognosticating. But it seems that the press, by going on to insist that Silver and his ilk are the new magic prognosticators, are missing the point just as much as those who thought the election could be predicted by political pundits.
Statistics is a tool for highlighting the probabilities. I'm sure that Nate Silver clones are going to be appearing a lot more on TV during the next major election cycles -- and I think that's a step forward. But now it seems like some people are expecting Silver and other stats guys to be right every time. And that's going to lead to backlash, just as the "failure" of Moneyball-type analysis to always get it exactly right resulted in some backlash in baseball. There will be data analysis in future election cycles -- likely from Silver himself -- that is wrong. That's the nature of probabilities. It will happen. And, unfortunately, people will then suddenly go back to arguing the opposite: that the stats geeks were "wrong."
But, as they say in the stats world, these are small sample size issues. Believing that statistical analysis is a perfect tool for predictions based on a single election is almost (though not quite) as weak as some of the traditional political punditry methods for predictions.
Hopefully, as with baseball, after a few years, the whole idea that these are entirely separate worlds will melt away. In baseball, every team now uses detailed statistical analysis as a tool, and most seem to understand that it suggests probabilities that help them find underexploited opportunities. But no one relies on it as a crystal ball that predicts the absolute future. Hopefully we'll reach that same sort of equilibrium in political analysis as well.