Sting Operation Shows How Full Of Crap Health Journals Are When It Comes To Dietary Studies

from the eat-it dept

Look, I probably don’t have to tell you Techdirt readers this, but I’m a strange sort of cat. I could go into all the reasons why I’m odd, but whenever I try to explain to people how non-normal I am, I usually just reveal this little bit of truth: I hate chocolate. No, I don’t not-love chocolate. Nor do I dislike chocolate. I fucking hate it, nearly as much as I hate how low I appeared on this ingenious bit of sleuthing a commenter did in determining which Techdirt writers swear the most (a list which I insist is fucking bullshit, by the way). That said, everyone else loves chocolate, of course, so I’m sure they and many others were thrilled to see so many well-respected publications blaring headlines recently about how chocolate can help reduce weight. I’d show you a bunch of links to those stories put forth by supposedly well-respected journalism outlets and scientific journals that make heavy claims about peer-reviews and fact-checking, but I can’t because most of those stories have been pulled. Why?

Because the whole thing was a bullshit hoax put on by a journalist to make the point that, at least when it comes to studies around diet and health, the journals and the media the reports on their papers are largely full of crap. Go read that entire thing, because it’s absolutely fascinating, but I’ll happily give you the truncated version. John Bohannon, who has a Ph.D in molecular biology of bacteria and is also a journalist, conspired with a German reporter, Peter Onneken, to see how badly they could fool the media to create BS headlines. They did this by turning John Bohannon into Johannes Bohannon (obviously) and creating a website for The Institute of Diet and Health, which isn’t actually a thing. Then they conducted a very real study with three groups: 1 group eating a low-carb diet, 1 group eating their regular diet, and 1 group eating a low-carb diet and a 1.5oz bar of dark chocolate daily. After running background on the groups, conducting blood tests to correct for disease and eating disorders, and hiring a German doctor and statistician to perform the study, away they went. The results?

Onneken then turned to his friend Alex Droste-Haars, a financial analyst, to crunch the numbers. One beer-fueled weekend later and… jackpot! Both of the treatment groups lost about 5 pounds over the course of the study, while the control group’s average body weight fluctuated up and down around zero. But the people on the low-carb diet plus chocolate? They lost weight 10 percent faster. Not only was that difference statistically significant, but the chocolate group had better cholesterol readings and higher scores on the well-being survey.

Bam, results! Not just results, but results the media would absolutely love to sink their idiotic teeth into. The problem? Well, the method for running the entire study was bullshit.

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.

Bohannon goes into some of the gory math, and it really is fun to read, but this is pretty easy to understand. With a small enough sample size and testing for as wide a range of results and factors as possible, you absolutely expect to find greater variance than if your study was testing for less factors or had a higher sample size. It’s simple: people are different and testing less people makes those difference statistically appear to be more significant.

Anyway, the team then went to the International Archives of Medicine, which Bohannon identifies as a “fake journal” publisher. In other words, pay enough Euros and your “study” gets “published”, all without the bothersome time-waster known as being peer-reviewed. Not that IAM doesn’t claim to be reviewed. It certainly does make that claim, but after payment was accepted Bohannon found that their study had been accepted without change. And, keep in mind, this study is designed to be bad. So, once the study had been published, it was time for the PR machine to swing into action.

Take a look at the press release I cooked up. It has everything. In reporter lingo: a sexy lede, a clear nut graf, some punchy quotes, and a kicker. And there’s no need to even read the scientific paper because the key details are already boiled down. I took special care to keep it accurate. Rather than tricking journalists, the goal was to lure them with a completely typical press release about a research paper. (Of course, what’s missing is the number of subjects and the minuscule weight differences between the groups.) But a good press release isn’t enough. Reporters are also hungry for “art,” something pretty to show their readers. So Onneken and Löbl shot some promotional video clips and commissioned freelance artists to write an acoustic ballad and even a rap about chocolate and weight loss. (It turns out you can hire people on the internet to do nearly anything.)

Onneken wrote a German press release and reached out directly to German media outlets. The promise of an “exclusive” story is very tempting, even if it’s fake. Then he blasted the German press release out on wire service based in Austria, and the English one went out on NewsWire. There was no quality control. That was left to the reporters.

And it didn’t take the reporters long to pick up this crap-on-a-stick and run with it like children after the ice cream truck. Not all of them, but some of the stories are still up. The Daily Star covered their paper, for instance, as did the Times of India, international editions of The Huffington Post, and some television news programs. Men’s Health was going to go with a story in September, though that probably won’t run now. Shape Magazine didn’t get off so lucky, with their story appearing in the June issue, in print. And remember, this is all bullshit. None of it is real. How does something like this happen?

The answer is lazy “journalists.”

When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted. These publications, though many command large audiences, are not exactly paragons of journalistic virtue. So it’s not surprising that they would simply grab a bit of digital chum for the headline, harvest the pageviews, and move on. But even the supposedly rigorous outlets that picked the study up failed to spot the holes.

Now, there is some humor in all of this, but also danger. It’s one thing to claim that chocolate leads to weight loss and have the media run wild with it, but we all know that fad diets and exciting health claims rain down on us in buckets, and I think it’s safe to say that not all of them are as harmless as Bohannon’s. The average person hasn’t done much thinking about the validity of these studies that they read about in the media; they simply trust the media to do the fact-checking. The media, it appears, largely trusts the journals to do the reviews and fact-checking. Except some (many?) of those journals don’t. The whole thing harkens back to one of the funnier moments in the Anchorman movie, when the main character makes a ludicrous claim about women’s brains being smaller than men’s, and then punctuates the statement with a smirk, saying, “It’s science.” As far as much of the media reporting goes, it might as well be “science.”

Strangely, Bohannon notes that readers of the articles were apparently more skeptical than the authors.

There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.

“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines. Or as one prescient reader of the 4 April story in the Daily Express put it, “Every day is April Fool’s in nutrition.”

If we’ve reached a time when readers are more skeptical than the reporters, that’s a massive problem for journalism, but perhaps a delightful sign for the spread of skepticism and inquiry amongst the public. Either way, look with a critical eye the next time you hear about that fad diet or health food claim.

Filed Under: , , , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Sting Operation Shows How Full Of Crap Health Journals Are When It Comes To Dietary Studies”

Subscribe: RSS Leave a comment
Anonymous Anonymous Coward says:

Strange Odd and Non-normal

Timothy, thank you for being strange, odd, and non-normal. I hope if your mother reads your pieces that she has spent a lot of time at truck stops and is therefore not offended.

But seriously, I need to talk to you about keeping hollandaise in the refrigerator when it is so easy to make fresh. Just [insert favorite search engine here] for blender hollandaise, it literally takes minutes and has to be better than anything that comes with whatever is necessary to make it stable. Use unsalted butter.

Oh, and having read the headlines for stories about the ‘chocolate’ diet (I could not bring myself to reading the stories) I thought about all the sugar added to most chocolate and cried ‘bullshit’ to myself.

Will-INI says:

When I saw the headline, I was hoping for an actual sting operation on some health journals. Instead I got some bullshit. This fake paper is just another example of what John P. A. Ioannidis’s documented in “Why Most Published Research Findings Are False.” It isn’t that more fake than actual papers that are not fake (but are still mostly or sorta fake).

Anonymous Coward says:

Re: Re:

The method on its own should not be used as specific proof of an effect, but as an indication requiring further researh. But because these studies are so much easier to make and understand than the determination of significant factors contributing, delving into the complex determination of how a body processes the main chemicals in chokolade and determining the associated chain of signal compounds, energy systems and many other effects, these black-box studies have become the norm.
His study is dime a dusin and it follows the norms for how to do these experiments, besides the conscious design flaws. Cudos to Bohannon for getting it as right as he did besides the conscious errors. It puts him above most in the field.

As for his conscious errors:
Measuring 18 parameters is crazy. Afaik the rule of thumb is 5 or 6 where you start entering the area of fitting an elephant.
15 testpersons is crazy low in any test. Usually you estimate the natural variance of your study and take in a few extra people for safety.
Calories and physical activity are very often the determining factors in weight loss. Not counting calories is the biggest tell that something is fishy in the study.

Anonymous Coward says:

Re: Re: Re:

“Not counting calories is the biggest tell that something is fishy in the study.”

First of all, the mythical calorie is a gross oversimplification of complex processes and is an erroneous unit of measure given the fact that everyone’s bodies process different types of foods in vastly different ways.

In other words, what might go straight through me might go straight to your gut and may even kill another if they’re allergic…

TKnarr (profile) says:

Not just diet and health

Here’s a list of papers appearing in chemistry journals:

To give you an idea, here’s titles for some of the papers:
JACS: “Science Rejected It, and Angewandte Couldn’t Think Up a Bad Enough Joke, So Here We Are”
Ang. Chem.: “A Metal-Organic Framework With Nanostructured BODIPY Ligands, Published Without Review on the Basis of the Title Alone”
J. Med. Chem.: “This Project Looks Good, But It Did Not Work. And 18 Out of the 23 Authors have Typographical Symbols Behind Their Names, Because The Work Took Place During Bush’s First Term”

David says:

The actually scary thing is:

They did not even falsify results. They just designed their study in a way that they were reasonably sure would deliver some false positives.

So they did not even sell wrong results but rather wrong conclusions.

Their point was not that the science establishment has weak defenses against frauds and charlatans. It was that it is has weak defenses against fools.

Kal Zekdor (profile) says:

Re: Re: Re:

Ehhh, I guess I half agree with you. While that specific piece was certainly a low-point for Ars (though it was an Op-Ed by a single author, if that makes a difference), it was also fairly thoroughly and uniformly blasted by the readers. None of which were banned for doing so.

[Side note: Do I want to know what a bronie (brony?) is?]

Weird Al is my Homeboy says:

I personall have been on an all chocolate

diet for 5 years now and it has helped me to go from a scrawny, pasty 135lbs to a healthy normal 437lbs so your wrong.

People used to kick sand in my face at the beach, making me unpopular and ridiculed, now they can’t kick it high enough to reach my face when I am laying down taking in the sun, my social life has exploded!.

Mark Gisleson (profile) says:


There is a weirdly effective guerrilla marketing scam the right uses. They see where the counterculture is making inroads, note how that information is being distributed, and then they start coming up with “stings” like this one that make you doubt the good information.

This chocolate study may have been about how easily duped we are, but it’s getting wide distribution because it makes people doubt the alternative health information out there. I saw this article in real time and dismissed it because it was so obviously indebted to wishful thinking.

Voluminous amounts of new information about diet are coming out. Much better documented and not fad dieting oriented. Simply put, we’ve been propagandized into ruining our diets. Why? Because you can make more money from processed foods. But hey, there was a phony story about chocolate, so you’d be stupid to read any of that stuff, which is why you’re going to continue to read about this study for a long, long time.

Rekrul says:

Look, I probably don’t have to tell you Techdirt readers this, but I’m a strange sort of cat. I could go into all the reasons why I’m odd, but whenever I try to explain to people how non-normal I am, I usually just reveal this little bit of truth: I hate chocolate. No, I don’t not-love chocolate. Nor do I dislike chocolate. I fucking hate it, nearly as much as I hate how low I appeared on this ingenious bit of sleuthing a commenter did in determining which Techdirt writers swear the most (a list which I insist is fucking bullshit, by the way).

People think I’m strange because I hate Summer.

Sure, the trees are all green and look nice, and the sunny sky looks cheerful, but that sunny sky also brings temperatures of 80-90 degrees. On a “cooler” day, a fan might be enough to keep you comfortable while inside, but the second you step outside, you start to sweat. Your hair is wet, your clothes are wet, you smell no matter much deodorant you’ve doused yourself with. On really hot days, a fan just pushes around hot air and you need an air conditioner to keep from leaving sweat stains on everything you touch. Of course running the air conditioner pretty much doubles your electric bill and walking outside is like stepping into a blast furnace.

Am I the only one who doesn’t enjoy sweating, or am I just the only person on the planet who sweats at temperatures above 74 degrees?

AJC says:

One misunderstanding here...

If one of the points of this post is to educate people on statistics, let me correct a comment that I’ve seen several times. It is not true that a smaller sample size increases the chance of a false positive (finding an effect when there isn’t one). Statistical tests are designed to account for sample size when calculating probabilities. The general idea is that a smaller sample size means you have to find a bigger effect before you can say it’s statistically significant (i.e. supposedly real), so that chance of getting a false positive is held constant regardless of sample size. This does mean, though, that an false positive will probably appear to have a larger effect with a small sample size than it would with a larger sample size. And it is also true (not surprisingly) that the chance of a false negative is larger with a smaller sample size.

Kal Zekdor (profile) says:


I can’t say that I hate chocolate, but I certainly dislike it, as well as anything sweet. Chocolate isn’t sweet enough to make me hate it, and dark chocolate’s not bad in small quantities (the more bitter, the better). Though, I saw a friend eat a Cadbury Creme Egg a few months ago, and nearly vomited as a result… How can people stand that? Yet, somehow, I’m the odd one…

W. Vann Hall (profile) says:

Not just for-pay health journals

This article from ‘JAMA Psychiatry’ makes all sort of expansive claims from a tiny, self-selected study group. It illustrates the problem of giving access to a raft of analysis tools but without common-sense training on when and how to use them. (Example: They ‘measure’ dose effects based on their estimate of ‘typical’ dose size for a street drug notorious for containing little or none of the purported active ingredient, often in combination with a raft of other substances, times the number of past drug sessions experiences by the participant. *Then* they do the same for an array of other street drugs in order to compare lifetime dosage effects. Oh, for a population all aged 18 to 25.)

Scott says:

Is it really the journalists fault?

Is this really the fault of the journalists? A scientist created a study, and a peer reviewed journal published it. Is it the job of Journalists to “catch” the bad science? Isn’t that the job of the peer review process?

While this is a blatant example, most examples of this phenomena will not include obvious problems that a journalist could have spotted.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...