Sting Operation Shows How Full Of Crap Health Journals Are When It Comes To Dietary Studies
from the eat-it dept
Look, I probably don’t have to tell you Techdirt readers this, but I’m a strange sort of cat. I could go into all the reasons why I’m odd, but whenever I try to explain to people how non-normal I am, I usually just reveal this little bit of truth: I hate chocolate. No, I don’t not-love chocolate. Nor do I dislike chocolate. I fucking hate it, nearly as much as I hate how low I appeared on this ingenious bit of sleuthing a commenter did in determining which Techdirt writers swear the most (a list which I insist is fucking bullshit, by the way). That said, everyone else loves chocolate, of course, so I’m sure they and many others were thrilled to see so many well-respected publications blaring headlines recently about how chocolate can help reduce weight. I’d show you a bunch of links to those stories put forth by supposedly well-respected journalism outlets and scientific journals that make heavy claims about peer-reviews and fact-checking, but I can’t because most of those stories have been pulled. Why?
Because the whole thing was a bullshit hoax put on by a journalist to make the point that, at least when it comes to studies around diet and health, the journals and the media the reports on their papers are largely full of crap. Go read that entire thing, because it’s absolutely fascinating, but I’ll happily give you the truncated version. John Bohannon, who has a Ph.D in molecular biology of bacteria and is also a journalist, conspired with a German reporter, Peter Onneken, to see how badly they could fool the media to create BS headlines. They did this by turning John Bohannon into Johannes Bohannon (obviously) and creating a website for The Institute of Diet and Health, which isn’t actually a thing. Then they conducted a very real study with three groups: 1 group eating a low-carb diet, 1 group eating their regular diet, and 1 group eating a low-carb diet and a 1.5oz bar of dark chocolate daily. After running background on the groups, conducting blood tests to correct for disease and eating disorders, and hiring a German doctor and statistician to perform the study, away they went. The results?
Onneken then turned to his friend Alex Droste-Haars, a financial analyst, to crunch the numbers. One beer-fueled weekend later and… jackpot! Both of the treatment groups lost about 5 pounds over the course of the study, while the control group’s average body weight fluctuated up and down around zero. But the people on the low-carb diet plus chocolate? They lost weight 10 percent faster. Not only was that difference statistically significant, but the chocolate group had better cholesterol readings and higher scores on the well-being survey.
Bam, results! Not just results, but results the media would absolutely love to sink their idiotic teeth into. The problem? Well, the method for running the entire study was bullshit.
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
Bohannon goes into some of the gory math, and it really is fun to read, but this is pretty easy to understand. With a small enough sample size and testing for as wide a range of results and factors as possible, you absolutely expect to find greater variance than if your study was testing for less factors or had a higher sample size. It’s simple: people are different and testing less people makes those difference statistically appear to be more significant.
Anyway, the team then went to the International Archives of Medicine, which Bohannon identifies as a “fake journal” publisher. In other words, pay enough Euros and your “study” gets “published”, all without the bothersome time-waster known as being peer-reviewed. Not that IAM doesn’t claim to be reviewed. It certainly does make that claim, but after payment was accepted Bohannon found that their study had been accepted without change. And, keep in mind, this study is designed to be bad. So, once the study had been published, it was time for the PR machine to swing into action.
Take a look at the press release I cooked up. It has everything. In reporter lingo: a sexy lede, a clear nut graf, some punchy quotes, and a kicker. And there’s no need to even read the scientific paper because the key details are already boiled down. I took special care to keep it accurate. Rather than tricking journalists, the goal was to lure them with a completely typical press release about a research paper. (Of course, what’s missing is the number of subjects and the minuscule weight differences between the groups.) But a good press release isn’t enough. Reporters are also hungry for “art,” something pretty to show their readers. So Onneken and Löbl shot some promotional video clips and commissioned freelance artists to write an acoustic ballad and even a rap about chocolate and weight loss. (It turns out you can hire people on the internet to do nearly anything.)
Onneken wrote a German press release and reached out directly to German media outlets. The promise of an “exclusive” story is very tempting, even if it’s fake. Then he blasted the German press release out on wire service based in Austria, and the English one went out on NewsWire. There was no quality control. That was left to the reporters.
And it didn’t take the reporters long to pick up this crap-on-a-stick and run with it like children after the ice cream truck. Not all of them, but some of the stories are still up. The Daily Star covered their paper, for instance, as did the Times of India, international editions of The Huffington Post, and some television news programs. Men’s Health was going to go with a story in September, though that probably won’t run now. Shape Magazine didn’t get off so lucky, with their story appearing in the June issue, in print. And remember, this is all bullshit. None of it is real. How does something like this happen?
The answer is lazy “journalists.”
When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted. These publications, though many command large audiences, are not exactly paragons of journalistic virtue. So it’s not surprising that they would simply grab a bit of digital chum for the headline, harvest the pageviews, and move on. But even the supposedly rigorous outlets that picked the study up failed to spot the holes.
Now, there is some humor in all of this, but also danger. It’s one thing to claim that chocolate leads to weight loss and have the media run wild with it, but we all know that fad diets and exciting health claims rain down on us in buckets, and I think it’s safe to say that not all of them are as harmless as Bohannon’s. The average person hasn’t done much thinking about the validity of these studies that they read about in the media; they simply trust the media to do the fact-checking. The media, it appears, largely trusts the journals to do the reviews and fact-checking. Except some (many?) of those journals don’t. The whole thing harkens back to one of the funnier moments in the Anchorman movie, when the main character makes a ludicrous claim about women’s brains being smaller than men’s, and then punctuates the statement with a smirk, saying, “It’s science.” As far as much of the media reporting goes, it might as well be “science.”
Strangely, Bohannon notes that readers of the articles were apparently more skeptical than the authors.
There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.
“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines. Or as one prescient reader of the 4 April story in the Daily Express put it, “Every day is April Fool’s in nutrition.”
If we’ve reached a time when readers are more skeptical than the reporters, that’s a massive problem for journalism, but perhaps a delightful sign for the spread of skepticism and inquiry amongst the public. Either way, look with a critical eye the next time you hear about that fad diet or health food claim.
Filed Under: chocolate, fooling journalists, hoax, journalism, science, statistics, studies, weight loss
Comments on “Sting Operation Shows How Full Of Crap Health Journals Are When It Comes To Dietary Studies”
LOLZ
Strange Odd and Non-normal
Timothy, thank you for being strange, odd, and non-normal. I hope if your mother reads your pieces that she has spent a lot of time at truck stops and is therefore not offended.
But seriously, I need to talk to you about keeping hollandaise in the refrigerator when it is so easy to make fresh. Just [insert favorite search engine here] for blender hollandaise, it literally takes minutes and has to be better than anything that comes with whatever is necessary to make it stable. Use unsalted butter.
Oh, and having read the headlines for stories about the ‘chocolate’ diet (I could not bring myself to reading the stories) I thought about all the sugar added to most chocolate and cried ‘bullshit’ to myself.
Re: Strange Odd and Non-normal
I did see it on a food show (at work at lunch, so I have no idea what show it is), and the experts on there actually said, “If chocolate DOES have any effect, the high cacao percentage, low sugar versions would be the ones that do. Any chocolate that’s full of sugar will not help you lose weight.”
” I fucking hate it”
Unfortunaltely for you, “Life is like a box of chocolates”.
News Rigging is here...Bot News..
News Rigging has arrived and how many of the studies/articles were written by bots…60% of the news today, written by bot and there’s a lot of fake grass root campaigns and Astro Turfing going on, be a skeptic with the news.
http://ducknetweb.blogspot.com/
Re: News Rigging is here...Bot News..
News Rigging has arrived and how many of the studies/articles were written by bots…
It’s strange to complain about news written by bots when the problem in this case is news written by people. Or are you claiming these stories about the chocolate study were not in fact written by people?
Re: News Rigging is here...Bot News..
The above was written by a bot, btw.
I like eating Lipton chicken noodle cup-a-soup dry, so there Mr. I hate chocolate.
Re: Re:
There are two things I like about this site. The articles are both humourous (at times) and educational, as well as the comments.
I just about choked on my Hershey bar dipped in Lipton chicken noodle cup-a-soup when I read it.
LOL
Re: Re:
I like to eat these dry;
http://forthemommas.com/wp-content/uploads/2015/05/Hungry-Jack-Hashbrown-Potatoes.jpg
Or after I’ve added the water, but before they’re cooked.
Re: I like eating Lipton chicken noodle cup-a-soup dry, so there Mr. I hate chocolate
Yum….crunchy and delicious!
Good article. I love hearing about people getting busted for sleeping on the job.
That said, Timothy please, for the sake of readability look up and embrace the word ‘fewer’. Please?
“if your study was testing for less factors or had a higher sample size. It’s simple: people are different and testing less people”
Re: "less" versus "fewer"
Interesting fact: mistakenly using “less” in lieu of “fewer” is a common symptom of chocolate deficiency. I read about it in the news.
Re: Re: "less" versus "fewer"
It’s science.
/smirk
Re: Re:
read this to learn about less and fewer
Re: Re: Re:
^http://languagelog.ldc.upenn.edu/nll/index.php?s=less+fewer
Re: Re:
Yay! These days, “less” is usually, and incorrectly, used where “fewer” is the correct usage. Always rubs me the wrong way and is usually a big, noisy tip-off that the user wasn’t paying attention somewhere along the way… or, maybe, his teachers didn’t know either.
When I saw the headline, I was hoping for an actual sting operation on some health journals. Instead I got some bullshit. This fake paper is just another example of what John P. A. Ioannidis’s documented in “Why Most Published Research Findings Are False.” It isn’t that more fake than actual papers that are not fake (but are still mostly or sorta fake).
Re: Re:
The method on its own should not be used as specific proof of an effect, but as an indication requiring further researh. But because these studies are so much easier to make and understand than the determination of significant factors contributing, delving into the complex determination of how a body processes the main chemicals in chokolade and determining the associated chain of signal compounds, energy systems and many other effects, these black-box studies have become the norm.
His study is dime a dusin and it follows the norms for how to do these experiments, besides the conscious design flaws. Cudos to Bohannon for getting it as right as he did besides the conscious errors. It puts him above most in the field.
As for his conscious errors:
Measuring 18 parameters is crazy. Afaik the rule of thumb is 5 or 6 where you start entering the area of fitting an elephant.
15 testpersons is crazy low in any test. Usually you estimate the natural variance of your study and take in a few extra people for safety.
Calories and physical activity are very often the determining factors in weight loss. Not counting calories is the biggest tell that something is fishy in the study.
Re: Re: Re:
“Not counting calories is the biggest tell that something is fishy in the study.”
First of all, the mythical calorie is a gross oversimplification of complex processes and is an erroneous unit of measure given the fact that everyone’s bodies process different types of foods in vastly different ways.
In other words, what might go straight through me might go straight to your gut and may even kill another if they’re allergic…
Re: Re: Re: Re:
given the fact that everyone’s bodies process different types of foods in vastly different ways.
That’s why you test with lots of people and look at specifics like standard deviation rather than just one number.
And people call me a denialist and a contrarian for being skeptical of everything I read or hear…It’s crap like this that constantly reminds of why I’m always so skeptical.
Not just diet and health
Here’s a list of papers appearing in chemistry journals:
http://pipeline.corante.com/archives/2015/05/12/a_glance_across_the_literature.php
To give you an idea, here’s titles for some of the papers:
JACS: “Science Rejected It, and Angewandte Couldn’t Think Up a Bad Enough Joke, So Here We Are”
Ang. Chem.: “A Metal-Organic Framework With Nanostructured BODIPY Ligands, Published Without Review on the Basis of the Title Alone”
J. Med. Chem.: “This Project Looks Good, But It Did Not Work. And 18 Out of the 23 Authors have Typographical Symbols Behind Their Names, Because The Work Took Place During Bush’s First Term”
The actually scary thing is:
They did not even falsify results. They just designed their study in a way that they were reasonably sure would deliver some false positives.
So they did not even sell wrong results but rather wrong conclusions.
Their point was not that the science establishment has weak defenses against frauds and charlatans. It was that it is has weak defenses against fools.
Re: The actually scary thing is:
The study mimics, deliberately, the fact that too many medical studies are done using excessively small sample sizes. This means there is a tendency for false positive and negatives. Also, they are chastising the numerical illiteracy of the press who do not even try to understand statistics.
HEY!
Learn to use “less” and “fewer” correctly. You’ll make fewer mistakes and sound less like a chocolate-hating, foul-mouthed Prenda lawyer.
Who saw the ArsTechnica article about this? http://arstechnica.com/science/2015/05/is-it-okay-to-generate-a-fake-news-story-to-make-a-point-no/
So, a bunch of lazy journalists’ laziness came back to bite them in the ass. Their response? “Wah! Those dirty scientists tricked us! They can’t experiment on humans!” Nice.
Re: Re:
ArsTechnica has got to be the most hypocritical science and technology site out there…And don’t even get me started on their reader base which seems to mainly consist of bronies…Any rebuttals that go against their narratives will instantly get you banned. Pfft
Re: Re: Re:
Ehhh, I guess I half agree with you. While that specific piece was certainly a low-point for Ars (though it was an Op-Ed by a single author, if that makes a difference), it was also fairly thoroughly and uniformly blasted by the readers. None of which were banned for doing so.
[Side note: Do I want to know what a bronie (brony?) is?]
Re: Re: Re: Re:
it was also fairly thoroughly and uniformly blasted by the readers
Also other Ars authors. The very first comment taking issue with some points in the article is an Ars author.
A book on this subject
The book “Statistics Done Wrong” by Alex Reinhart (No Starch Press) covers this (without any math). All too often studies are designed without considering the sample size required to reduce false positive and negative rates to a small number and the fact that most papers do not publish the full statistical treatment.
I personall have been on an all chocolate
diet for 5 years now and it has helped me to go from a scrawny, pasty 135lbs to a healthy normal 437lbs so your wrong.
People used to kick sand in my face at the beach, making me unpopular and ridiculed, now they can’t kick it high enough to reach my face when I am laying down taking in the sun, my social life has exploded!.
Re: I personall have been on an all chocolate
Grendel?
Careful
There is a weirdly effective guerrilla marketing scam the right uses. They see where the counterculture is making inroads, note how that information is being distributed, and then they start coming up with “stings” like this one that make you doubt the good information.
This chocolate study may have been about how easily duped we are, but it’s getting wide distribution because it makes people doubt the alternative health information out there. I saw this article in real time and dismissed it because it was so obviously indebted to wishful thinking.
Voluminous amounts of new information about diet are coming out. Much better documented and not fad dieting oriented. Simply put, we’ve been propagandized into ruining our diets. Why? Because you can make more money from processed foods. But hey, there was a phony story about chocolate, so you’d be stupid to read any of that stuff, which is why you’re going to continue to read about this study for a long, long time.
People think I’m strange because I hate Summer.
Sure, the trees are all green and look nice, and the sunny sky looks cheerful, but that sunny sky also brings temperatures of 80-90 degrees. On a “cooler” day, a fan might be enough to keep you comfortable while inside, but the second you step outside, you start to sweat. Your hair is wet, your clothes are wet, you smell no matter much deodorant you’ve doused yourself with. On really hot days, a fan just pushes around hot air and you need an air conditioner to keep from leaving sweat stains on everything you touch. Of course running the air conditioner pretty much doubles your electric bill and walking outside is like stepping into a blast furnace.
Am I the only one who doesn’t enjoy sweating, or am I just the only person on the planet who sweats at temperatures above 74 degrees?
Re: Re:
You gotta get yourself some of those heat deflecting blinds and tint your windows. Granted it only reduces the heat by 5-10% – but for me it’s just enough to get away with only a fan…And these are on days with 100% humidity at 90° fahrenheit which isn’t half bad once it’s all setup.
Re: Re:
Am I the only one who doesn’t enjoy sweating, or am I just the only person on the planet who sweats at temperatures above 74 degrees?
I think it’s more that a lot of people really really hate cold weather, so they would rather put up with heat than cold. But that’s just a guess.
Re: Re: Re:
I’ll take cold weather over hot any day. If it’s too cold, you can just wear more. Too hot, and there’s only so many layers you can take off.
Money compromises all.
One misunderstanding here...
If one of the points of this post is to educate people on statistics, let me correct a comment that I’ve seen several times. It is not true that a smaller sample size increases the chance of a false positive (finding an effect when there isn’t one). Statistical tests are designed to account for sample size when calculating probabilities. The general idea is that a smaller sample size means you have to find a bigger effect before you can say it’s statistically significant (i.e. supposedly real), so that chance of getting a false positive is held constant regardless of sample size. This does mean, though, that an false positive will probably appear to have a larger effect with a small sample size than it would with a larger sample size. And it is also true (not surprisingly) that the chance of a false negative is larger with a smaller sample size.
Re: One misunderstanding here...
So what happens when you you introduce a populace that are incredibly closely related (such as Mongolians and Icelanders)and mix them with otheres….bogus results no matter witch way you frame it.
Chocolate
I can’t say that I hate chocolate, but I certainly dislike it, as well as anything sweet. Chocolate isn’t sweet enough to make me hate it, and dark chocolate’s not bad in small quantities (the more bitter, the better). Though, I saw a friend eat a Cadbury Creme Egg a few months ago, and nearly vomited as a result… How can people stand that? Yet, somehow, I’m the odd one…
Not just for-pay health journals
This article from ‘JAMA Psychiatry’ makes all sort of expansive claims from a tiny, self-selected study group. It illustrates the problem of giving access to a raft of analysis tools but without common-sense training on when and how to use them. (Example: They ‘measure’ dose effects based on their estimate of ‘typical’ dose size for a street drug notorious for containing little or none of the purported active ingredient, often in combination with a raft of other substances, times the number of past drug sessions experiences by the participant. *Then* they do the same for an array of other street drugs in order to compare lifetime dosage effects. Oh, for a population all aged 18 to 25.)
http://archpsyc.jamanetwork.com/article.aspx?articleid=1151061
Is it really the journalists fault?
Is this really the fault of the journalists? A scientist created a study, and a peer reviewed journal published it. Is it the job of Journalists to “catch” the bad science? Isn’t that the job of the peer review process?
While this is a blatant example, most examples of this phenomena will not include obvious problems that a journalist could have spotted.
Re: Is it really the journalists fault?
Nope, as it states in the article it wasn’t published in a peer reviewed journal, this journal works by you paying cash to get your ‘study’ published, no questions asked, apart from ‘is your money real?’.
Then Again ...
… some consider this sort of thing unethical.
RIP
Journalism died years ago. What we have now in its place is strictly sensationalism.
less factors... less people... ?
FEWER factors
FEWER people
Chocolate hatred
When I was young, I loved chocolate, the darker the better. Over the years, I somehow came to the point where I dislike it. I don’t hate it (yet, my taste is continuing to evolve) to the point that I will simply refuse to eat it, but I’d rather pass.
“Either way, look with a critical eye the next time you hear about that fad diet or health food claim.”
Or any social sciences ‘study”.
” a ludicrous claim about women’s brains being smaller than men’s”
What is ludicrous about that claim? The science is pretty well established that female brains are on average about 10 % smaller.
Re: Re:
This is true. The ludicrous part is the implication that the size difference means anything at all about intelligence. It doesn’t, because that’s not how brains work. Size matters in certain contexts, but a 10% difference within the same species doesn’t make a difference.
ethics
What do you think of the ethical imications of Gawker, of all the sensationalist rags, trolling other journalism outlets? Myself, I’m confused about it. I shudder at their engagement creating the very sensationalism they report on- see “mein coke” and gamergate.
I can’t eat cheap chocolate. I really enjoy See’s peanut butter cups, I forget what they call them in the store.