Chocolate Still Won't Help You Lose Weight: How A Fake Scientific Study Fooled The Media
Chocolate can help you lose weight? It sounded too good to be true, and it was. John Bohannon, science journalist and molecular biologist, set out to prove the power of junk science by making up an official-sounding scientific institution, reporting the results of a study based on what he calls "terrible science," and authoring the aforementioned study with a false name and PhD (Johannes Bohannon instead of John, and a nutritional biology degree instead of a molecular bacteria biology degree). Many media outlets were fooled by the story.
Earlier this year, Bohannon created a website for something called the "Institute of Diet and Health" and submitted a study to the International Archives of Medicine, a peer-reviewed medical journal, claiming that "consumption of chocolate with a high cocoa content can significantly increase the success of weight-loss diets." The study was real — it compared three small groups of people: a control group that made no diet changes and two variable groups. One group consumed a low-carb diet, and another group consumed the same diet but added a 1.5-ounce bar of chocolate. After three weeks, the chocolate group lost weight 10 percent faster overall. But getting "scientific results" does not necessarily make a study accurate, claims Dr. Bohannon.
"Here's a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a statistically significant result," Dr. Bohannon wrote in a post on i09. "Our study included 18 different measurements — weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. — from 15 people. That study design is a recipe for false positives. We didn't know exactly what would pan out — the headline could have been that chocolate improves sleep or lowers blood pressure — but we knew our chances of getting at least one statistically significant result were pretty good."
The International Archives of Medicine agreed to publish the study without any semblance of a rigorous peer-review process, says Bohannon. After publication, as predicted, dozens of well-known publications were duped: Shape Magazine devoted a column to the study, while the Irish Examiner, Cosmopolitan's German publication, and The Daily Star also reported on it.
"If a study doesn't even list how many people took part in it, or makes a bold diet claim that's statistically significant but doesn't say how big the effect size is, you should wonder why," Bohannon explained. "But for the most part, we don't. Which is a pity, because when we fail, the world is awash in junk science."