(CBS News) Eating chocolate every day can help you lose weight? If it sounds too good to be true -- that's because the chocolate diet study that made headlines around the world last year was all an elaborate hoax.
Now those responsible are going public with the story behind the bogus diet study and the media frenzy that followed. It was a carefully planned effort to expose the prevalence of junk science and unchecked, hype-driven press coverage.
"The world is just drowning in all this pseudoscience" about diet and nutrition, science journalist John Bohannon, one of the collaborators in the project, told CBS News, "and when there is science, it's very poorly reported. We [journalists] should be doing a better job, and the only way to do it is to kind of shock the system."
In an article posted on the website io9.com, Bohannon explains how the chocolate diet story came about. He was first approached by a German TV producer, Peter Onneken, who was working with Diana Löbl and others on a documentary film about junk science. It's a topic Bohannon -- who has a Ph.D. in molecular biology -- has covered extensively. He previously conducted a sting operation, published in Science in 2013, exposing how some unscrupulous open-access journals would publish fake scientific studies for a fee without subjecting them to peer review.
Bohannon and the filmmakers concocted a plan to prove just how easy it is to turn bad science into big headlines. They created a website for the Institute of Diet and Health (a group they made up), recruited a doctor and analyst, and paid research subjects to take part in a small clinical trial they would run to test the effects of eating chocolate. Then Bohannon would use his media savvy to get the results published and publicized.
The doctor, Bohannon writes, was in on it from the beginning. Dr. Gunter Frank, a general practitioner in Germany, had previously written a book blasting pseudoscience in the diet industry.
"Testing bitter chocolate as a dietary supplement was his idea," Bohannon writes. "When I asked him why, Frank said it was a favorite of the 'whole food' fanatics. 'Bitter chocolate tastes bad, therefore it must be good for you,' he said. 'It's like a religion.'"
Onneken and his team ran it like a real clinical trial, with one-third of the subjects randomly put on a low-carb diet, one-third assigned to eat the same low-carb diet plus a 1.5-ounce bar of dark chocolate every day, and a control group told to follow their regular diets. The participants weighed in for 21 days, and at the end of the study, the analyst crunched the numbers. He found that the chocolate-eating group lost weight about 10 percent faster than the other dieters.
"I know what you're thinking," Bohannon writes. "The study did show accelerated weight loss in the chocolate group -- shouldn't we trust it? Isn't that how science works?"
Then he explains:
"Here's a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a 'statistically significant' result. Our study included 18 different measurements -- weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. -- from 15 people. (One subject was dropped.) That study design is a recipe for false positives....We didn't know exactly what would pan out -- the headline could have been that chocolate improves sleep or lowers blood pressure -- but we knew our chances of getting at least one 'statistically significant' result were pretty good."
So the team wrote up a report showcasing the results they liked, and went looking for a publisher. "Since it was such bad science, we needed to skip peer review altogether," Bohannon writes.
He sent the manuscript to some of the same sketchy journals he'd exposed in his sting operation, and several offered to run it as long as he paid a fee. Two weeks later, it appeared in the International Archives of Medicine, which Bohannon says didn't edit a word and charged 600 Euros (about $660) for the privilege.
Now that he'd exposed the underside of the scientific process, Bohannon and his collaborators went after the media. They wrote up buzzworthy press releases in English and German quoting "the study's lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health," which caught the eye of multiple news organizations.
The German newspaper Bild took the bait, and the Daily Mail and others quickly followed. "Soon we were in the Daily Star, the Irish Examiner, Cosmopolitan's German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show," Bohannon writes. Popular fitness magazines Shape and Prevention also featured the irresistible new diet advice. (Prevention has now posted an editor's note stating that the study was "conducted by an unqualified researcher who manipulated the statistics.")
Bohannon says few reporters asked any probing questions about how the study was conducted. No one seemed to have Googled him or questioned why the Institute of Diet and Health had no previous track record. "Hopefully our little experiment will make reporters and readers alike more skeptical," Bohannon writes.
He told CBS News he thinks the endless stream of diet stories promising quick fixes is cause for serious concern. "Obesity matters," he said. "It is unambiguous that people are suffering and dying because of obesity on a huge scale. We need to report on this."
So how does a reader know whether the scientific research behind a story is legit? Look for information like whether a study was conducted on humans or animals; whether a large number of people took part; the measurable significance of any effect the researchers claim to find; and whether it's published in a respected peer-reviewed journal (although that's not foolproof).
"Here is the first and easiest rule for a reader: If some news article seems to be giving you diet advice -- a bold claim like eating this or not eating that is good or bad -- you don't have to read further," Bohannon said. "The scientific consensus has not crystallized around diet and human health outcomes to the degree that you can make any claims yet. This is hard science. I'm afraid to say that the vast majority of reporting in this area -- almost everything you see in glossy magazines devoted to men's and women's health -- is bunk at worst or very, very poorly reported at best."
Bohannon noted that his goal was not to embarrass individual reporters who picked up the chocolate study -- he's got his eye on bigger targets. "I know the pressure they're under. The blame really rests with the editors and owners of these media outlets -- they're the ones who are profiting on this information, and they're pushing their reporters relentlessly with this daily grind of getting headlines out. You have to start by shaming them."
Real scientific work on nutrition and health is often incremental, inconclusive or contradictory, not the source of easy answers and sexy headlines -- no matter how much we may wish the chocolate diet really worked.
