You, Yes You, Are To Blame For Junk Stats

from the 9-out-of-10-people-disagree dept

It's pretty easy to find examples of bad statistics, whether they stem from poor data collection or interpretation. But a writer over at the BBC thinks he's fingered the real culprit: people who respond to surveys incorrectly. While on one level, his claim may have a little bit of truth to it, it really seems like an abdication of responsibility for the media, who all too often don't check out the bogus stats they cite, or swallow manipulated data without much question. Furthermore, when many groups use push polls not just to collect data, but to influence opinions, it's hard to blame the average respondent too much. In short, bad responses to survey questions could be a problem -- but that doesn't excuse the people collecting the data, and especially the media reporting it, from ignoring the issue. If they know the data's inaccurate, why do they keep reporting it?


Reader Comments (rss)

(Flattened / Threaded)

  1.  
    icon
    lovingseem (profile), Jun 4th, 2009 @ 9:31pm

    examples of Bad

    I've filed it along with examples of Bad Statistics because the issue is of chi-squared into a probability is not quite as easy as it may seem. Yes, we all know better now, but I don't think its one or the other. all those weight loss ads are junk you are what you eat.plain and simple,Thanks loving seem adult

     

    reply to this | link to this | view in thread ]

  2.  
    identicon
    inc, Jun 4th, 2009 @ 10:24pm

    if you are collecting bad stats it the fault of the person collecting the stats. I notice a lot of times surveys are forced upon you when you don't want to be filling one out. This is a perfect example of when you would get back answers that are way off base.

     

    reply to this | link to this | view in thread ]

  3.  
    identicon
    Anonymous Coward, Jun 4th, 2009 @ 11:53pm

    News flash: people get annoyed when you hassle them to do surveys they have no interest in. Of course they will answer with a load of BS. Of course if you just make it voluntary, only those with a vested interest will respond, and the results will be biased.
    The obvious outcome of all this?...Media should stop concocting and reporting these stats as if they actually mean something.

     

    reply to this | link to this | view in thread ]

  4.  
    identicon
    Andrew, Jun 5th, 2009 @ 12:41am

    Amazing spam

    Re:"examples of bad" - that is the most intelligent bit of comment spam I've seen in a long time. Is that autogenerated, do you think?

     

    reply to this | link to this | view in thread ]

  5.  
    icon
    Sam (profile), Jun 5th, 2009 @ 1:40am

    I should point out that this guy has been writing a column on bad statistics on the BBC News website for months, and is also the author of a rather good book on bad stats called The Tiger that Isn't. He's not citing people responding incorrectly as *the* cause of bad stats, merely one of the many many factors covered in his regular column.

     

    reply to this | link to this | view in thread ]

  6.  
    icon
    Tor (profile), Jun 5th, 2009 @ 1:54am

    Just use another poll to analyse the first one

    A couple of years ago a big Swedish newspaper ran an online poll on their web page with the following question: "Do you answer truthfully to web poll questions?"

    Enough said.

     

    reply to this | link to this | view in thread ]

  7.  
    identicon
    Anonymous Coward, Jun 5th, 2009 @ 3:27am

    Junk stats? Junk stats are bad, running with them like they are the word of god is worse.

    http://latimesblogs.latimes.com/technology/2009/06/bittorrent-vuze-survey-purchasing-dvd.h tml

    I think that is the best example of junk stats possible. Their survey group including almost 65% Vuse users, in a study published by, well, Vuse. Talk about misleading and meaningless junk stats. Yet Carlo, your boss ran with this story like it had come from on high, because it supports his (still unproven) concept that somehow people who download everything are still willing to pay for it, and more often than people who don't.

    Really, you should chat with Mike before you put these things up, because it would seem that the two of you just don't agree.

     

    reply to this | link to this | view in thread ]

  8.  
    identicon
    Laurent GUERBY, Jun 5th, 2009 @ 6:20am

    Unemployment

    One of the most commonly cited measure in economics is unemployment. But unemployment of 10% does mean that one out of ten person are without job, because of the concept of "active population" which is people with jobs plus people looking "actively" for a job. Each country typically defines 5 to 10 "actively looking" levels and publishes one as "official unemployment". OECD takes all those and publishes a normalized, that is comparable between countries, unemployment level. However a little know fact ...

    From OECD normalized numbers for 2007 (latest available):

    * men aged 25-54

    employment divided by population
    France: 88.3% (hence 11.7% jobless)
    USA: 87.5% (hence 12.5% jobless)

    normalized unemployment rate
    France: 6.3%
    USA: 3.7%

    So with a slightly higher relative working population in this age/sex group France has ... a 70% higher unemployment rate than the USA.

    * women aged 25-54

    E/P
    FR 76.1
    USA 72.5

    unemployment
    FR 7.7
    USA 3.8

    So here we have clearly more women working in France (3.6 percentage point) relative to their population but unemployment is 102% higher in France!

    * conclusion

    Employment/population is an objective measure. Unemployment is an extremely subjective measure and so it's easy to get very wide discrepancies as shown above.

    * question

    Any paper studying this discrepancy in OECD normalized numbers? For male 25-54 there's no real reason not to work, so what are all those USA men doing?

    * second question

    What credit do you place on "empirical validation" of economic theories (and papers) who use unemployment by country to select the right policy?

     

    reply to this | link to this | view in thread ]

  9.  
    icon
    EEJ (profile), Jun 5th, 2009 @ 7:58am

    Surveys

    Another problem with survey accuracy is the design of the surveys. I've taken several surveys online (in exchange for payment or entries in sweepstakes) and have been very frustrated when questions are asked but don't have enough viable answers available (or a way to explain your answer).

    There doesn't seem to be enough middle ground between some of the answers, and sometimes there isn't a viable answer avialable, or the question doesn't apply to me at all, based on previous answers I've given or my current situation.

     

    reply to this | link to this | view in thread ]

  10.  
    identicon
    Anonymous Coward, Jun 5th, 2009 @ 8:33am

    Carlo: You, Yes You, Are To Blame For Junk Posts

     

    reply to this | link to this | view in thread ]

  11.  
    identicon
    Ben, Jun 5th, 2009 @ 8:33am

    My fault

    I wonder how much I have affected advertising for our local paper. Because I refuse to divulge my information by creating an account, it frequently (every time I clear my cookies) asks me a few questions before I go to an article. I can just see the ad department trying to figure my habits as a female born in 1901 and living in zip code 90210.

     

    reply to this | link to this | view in thread ]

  12.  
    identicon
    porkster, Jun 7th, 2009 @ 1:00pm

    Guilty as charged!

    Yep,
    Thats me, giving out bad answers as usual.

    If you ring me in the middle of my dinner (actually if you ring me at all and I haven't requested it) of course your going to get a skewed results.

    If your web site insists on a pop-up for a survey and it keeps poping up then yes, your going to get cr*p answers.

    If you keep sending me e-mail to fill out a survey then you only have yourself to blame.

    And one guarenteed lost customer.

     

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This