John Bohannon the molecular biologist and science publishing gladfly, caused quite a stir last week with his story in iO9 boasting of having fooled millions with a widely reported, bogus study purportedly showing that adding a little bit of dark chocolate to a low carb diet accelerates weight loss. There were some grumblings of ethical concerns, mostly drowned in the round of applause by people happy to show how gullible health reporters and their readers can be when served up a story everyone would like to believe.
That was plenty of fun, but I think there are three takeaways worth taking a closer look at.
1. This chocolate study hoax reveals why so many of the studies that are used by fear-mongers and pseudo-scientists are not really studies, but statistical fishing expeditions.
Beyond just showing how bad nutritional reporting can be driven by trivial studies, this experiment showed how one of the great vehicles of pseudo-science and fear-mongering operates. For any scientist wanting to create evidence that politically contentious compound or ingredient is cause for concern, all they have to do is run a statistically under-powered trial and they are almost guaranteed to generate cannon fodder.
If you are ever trying to explain why say, … the infamous Séralini rat study can be dismissed out of hand, don’t point your friend to a thorough debunking of the Séralini study. No, point them to an admitted hoaxer explaining why a large number of variables among a small number of subjects is nearly guaranteed to generate any type of headline you want:
I know what you’re thinking. The study did show accelerated weight loss in the chocolate group—shouldn’t we trust it? Isn’t that how science works?
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
That’s the Séralini rat study in a nutshell, but it’s not going to be as polarizing as an actual debunking of Séralini. It’s also the model for any number of outlier “studies” that under-gird all sorts of various politically motivated chemophobia.
2. Beware of sciencey looking websites that just republish or regurgitate press releases.
You don’t need to avoid these sites, just know that what you are looking at. Research departments write their press releases to read like breezy 800 word stories on the research. There’s a reason for that. It doesn’t mean that the research is bad. It just mean that you are reading a press release. Phys.org is a great site that runs on press releases. But it looks like more than it is, just keep that in mind. Science Daily is another indispensable site for science news, but the name should tell you something about the pace that they are turning around stories. These sites are a great starting point. Nothing wrong with that.
3. Health reporters are not science reporters.
Though Bohannon referred to “science reporters” throughout the piece he finally comes clean near the end about what we are really talking about.
The only problem with the diet science beat is that it’s science. You have to know how to read a scientific paper—and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases.
With a few notable exceptions, (The New York Times, Men’s Health) health beat reporters are not science writers. They are to science writing what red carpet reporters are to film criticism.
[Please consider supporting FAFDL.org by making a donation to our library.]
I’ll leave you with Randall Monroe’s classic XKCD panel on the subject.
