Turns out this is not a healthy diet.

Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the ‘wet streets cause rain’ stories. Paper’s full of them.

In any case, you read with exasperation or amusement the multiple errors in a story—and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. — Michael Crichton (Why Speculate?)

According to John Bohannon, he fooled newspapers and magazines around the world into reporting on bad chocolate science. Now, given that he’s claiming to have lied to the world, it’s understandable if our first response is, why should we trust you now?

But his description of what they did makes perfect sense. Basically, they measured a whole bunch of things. By purely random chance, if you measure a whole bunch of things, some of your measurements will look like they’re significant even though, taken as a whole, they aren’t.

Think about it this way: if you toss one marble at a cup twenty feet away, it’s pretty impressive if you get it in. If you toss a hundred marbles at the cup, it’s no longer so impressive if one of them makes it in.

Bohannon basically ran the study as too many researchers do nowadays, just for the hell of it, then reported as if they had run a study for what they found afterward. This is a big, big mistake where randomness plays a part. When I was a young psychology student at Cornell, I was even taught never to run a study without knowing what, specifically, you were looking for and why. Even to find things to run a study about later.

That is, they could have run this study, discovered that among all the things they were measuring their chocolate subjects lost weight faster, and then designed a study around that finding to determine whether chocolate causes dieters to lose weight faster. That at least would have some significance about it.

But it still wouldn’t be science.

Science is: theory, prediction, test.

You can’t just go looking for things. You have to know why you are looking for that thing. You have to have a theory that predicts something. Then, from the combination of that theory and that prediction from the theory, you test that prediction.

You might say, I don’t need a theory of gravity to know that letting go of rocks means they’ll fall.

You’re right, you don’t. But that’s not science, or at least, it isn’t the scientific method.

Galileo had a theory, that things must fall at the same rate because otherwise you could connect things by strings and both sides of the connection would fall faster, just because they were connected. As theories go, that’s not Einstein-level, but it did allow a prediction: drop two different items from a tall height, and they should drop at the same acceleration. He tested that theory; the test matched his prediction.¹

It’s the gullible expression that identifies this as a journalist.

What John Bohannon describes is all too common in research today. Researchers want to know the “effects” of things. So they measure all sorts of “effects”. Sometimes they don’t even plan to measure an effect, but they gather the data as a side effect of some other measurement, and then that side effect turns out to be the only “significant” finding.

Significant here, however, is a misnomer. Statistical significance requires that you were looking specifically at that one effect and already predicted what form that effect would take. Otherwise, you’re relying, as Bohannon did, on chance to provide something that you can publish.

Further, because human testing is a pain in the ass—that’s a technical term—sample sizes are almost always smaller than we’d like anyway.

Without any idea of why you found what you did, there is no way for someone else to disprove your theory. Your theory was basically that if you run a whole bunch of tests, something interesting will pop up among that population. If someone else runs a similar test and comes up with different results, it doesn’t disprove your theory because you don’t have one.

The most important lesson, though, is that reporters are simply not intellectually equipped to sort out the good science from the bad. The news media is notoriously bad at getting even basic reporting right—the Gel-Mann amnesia effect is the most famous example—but the scientific method is an alien thing to most journalists. Good science is about numbers, replicability, and a systemic process. Journalists think in terms of anecdotes, black swans, and intentions.

Even Bohannon, whose writing is engaging, clear, and precise, gets it somewhat wrong at the end. After explaining why the shotgun approach makes a hash of experimental results, he then says:

If a study doesn’t even list how many people took part in it, or makes a bold diet claim that’s “statistically significant” but doesn’t say how big the effect size is, you should wonder why.

Those are important, especially the difference between statistically significant and practically significant. But he already identified the most important criteria in determining whether a study is “statistically significant”: how many effects they were looking at ahead of time vs. how many effects they reported on. He even went through the math to explain why most “significant” results are in fact nowhere near significant when taken in contest of everything else the researchers measured.

That is the one thing that requires shoe-leather reporting, because it will almost always be left out of the study’s press release even if sample size and size of the effect are presented.

Now if you’ll excuse me, I need to go over to Trader Joe’s and get some dark chocolate. Because I like it, not because it’s healthy.