Can Pundits Actually Prognosticate? Answer: Mostly, No

from the nice-to-see-some-lookback dept

I've discussed in the past how loath the tech industry often is to ever look back at analyst "predictions." The big research firms come out with all sorts of ridiculous predictions, and no one ever goes back and figures out how accurate they were. It seems that the same thing is frequently true with political prognosticators, so it's interesting to see a Hamilton College public policy class analyze the predictions of 26 political pundits over the period of 15 months (September 2007 to December 2008) and measure how good the pundits were. It turns out most were "no better than a coin toss." I'm not entirely convinced of the methodology, since it seems to make use of some subjective analysis from the description, but for the politically minded, it's at least interesting to note that the most "accurate" pundits all fall on the left of the traditional political spectrum, while the least successful tended to fall on the right. I do wonder how much of that has to do with the timing (the period covered the financial decline and the Presidential election). It would be interesting to see a similar test run during a different period of time as well. I wonder if a similar analysis, say, prior to the election of a Republican president, would have turned up the opposite results. In other words: was there a fundamental quality in the predictions, or was it just that "the winning team" looks smarter in retrospect? Either way, it's still great that people are going back and looking at how well some of these prognosticators did.


Reader Comments (rss)

(Flattened / Threaded)

  •  
    identicon
    John Doe, May 3rd, 2011 @ 11:28am

    The real lesson here...

    The real lesson here is don't listen to prognosticators.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 3rd, 2011 @ 11:29am

    Shouldn't "actually" be changed to "accurately". That would be more in keeping with the substance of the article.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    fogbugzd (profile), May 3rd, 2011 @ 11:29am

    I always figured that if an "expert" was correct more than 50% of the time, then it was a win. It is easy for experts to get distracted by some detail and miss the big picture which is why it is much easier to be wrong than right. I watched Bill Gates pretty closely when he was still running Microsoft, and it looked to me that he was right a little over half the time. But that was enough to make him a billionaire.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Thomas (profile), May 3rd, 2011 @ 11:33am

    "Reality has a well-know Liberal bias." - Colbert

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      John Doe, May 3rd, 2011 @ 12:23pm

      Re:

      "Liberalism is a mental disease." - forgot who said it. :)

       

      reply to this | link to this | view in chronology ]

      •  
        icon
        Steven (profile), May 3rd, 2011 @ 12:29pm

        Re: Re:

        "If you're not a liberal at twenty you have no heart, if you're not a conservative at forty you have no brain." - Winston Churchill

        And if you haven't figured out by now that neither party is actually looking out for you, you just can't be helped - Me

         

        reply to this | link to this | view in chronology ]

        •  
          identicon
          John Doe, May 3rd, 2011 @ 12:37pm

          Re: Re: Re:

          And if you haven't figured out by now that neither party is actually looking out for you, you just can't be helped - Me

          Very true. Unfortunately we only have the two parties to choose from so you have to pick one.

           

          reply to this | link to this | view in chronology ]

  •  
    icon
    Steven (profile), May 3rd, 2011 @ 11:37am

    Unskilled and Unaware

    I strongly suspect one of the major pitfalls of 'professional journalism' is exactly that they are trained in journalism. They then think this training grants them expertise in nearly all other areas of life. I would love to see a world in which journalism was only offered as a secondary major, and journalists reported on their primary fields.

    There is a fascinating paper (pdf) here:
    http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.2655&rep=rep1&type=pdf

    It shows how people tend to vastly overestimate their abilities in an area they are unskilled. If you've never read it you should.

    Of course seeing as how I'm unskilled in the field of journalism, it's entirely possible I've fallen into my own trap.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      ChurchHatesTucker (profile), May 3rd, 2011 @ 11:44am

      Re: Unskilled and Unaware

      I strongly suspect one of the major pitfalls of 'professional journalism' is exactly that they are trained in journalism. They then think this training grants them expertise in nearly all other areas of life.

      This is so true it hurts.

      We need to remember that almost all reporters are generalists, which means they're relying on 'sources' to feed them information. Just about anything you read from mainstream sources should be treated as a press release. Because it usually is.

       

      reply to this | link to this | view in chronology ]

    •  
      identicon
      Stuart, May 3rd, 2011 @ 12:59pm

      Re: Unskilled and Unaware

      Should be the other way around.
      Journalism should be the primary major.
      To report well on a subject you do not need intimate knowledge. Only need to have a firm grasp of how little you really know.

       

      reply to this | link to this | view in chronology ]

  •  
    icon
    A Dan (profile), May 3rd, 2011 @ 11:43am

    Typo

    "I've discussed in the past how loathe the tech industry often is"

    I think you mean "loath".

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 3rd, 2011 @ 12:20pm

    "The big research firms come out with all sorts of ridiculous predictions, and no one ever goes back and figures out how accurate they were."

    I predict the world will come to an end in 2012.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 3rd, 2011 @ 12:26pm

      Re:

      (and if this prediction doesn't come true, then I predict no one will ever go back to figure out how accurate my prediction was).

       

      reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This