The million-dollar question in nutrition science is this: What should we eat to live a long and healthy life?
Researchers’ answers to this question have often been contradictory and confusing. But in recent decades, one diet has attracted the lion’s share of research dollars and public attention: the Mediterranean way of eating. And in 2013, its scientific cred was secured with PREDIMED, one of the most important recent diet studies published.
The study’s delicious conclusion was that eating as the Spanish, Italian, and Greeks do — dousing food in olive oil and loading up on fish, nuts, and fresh produce — cuts cardiovascular disease risk by a third. As Stanford University health researcher (and nutrition science critic) John Ioannidis put it: “It was the best. The best of the best.”
Not anymore. Last week, the prestigious New England Journal of Medicine pulled the original paper from the record, issuing a rare retraction. It also republished a new version of PREDIMED, based on a reanalysis of the data that accounted for the missteps.
PREDIMED was supposed to be an example of scientific excellence in a field filled with conflicted and flawed studies. Yet it now appears to be horribly flawed.
At first, we thought this could be the beginning of the end of nutrition science. There have been too many poorly executed and disappointing studies over the years, too many research dollars wasted. (We’ve also just learned about industry influence in the National Institutes of Health’s alcohol studies, and infighting that brought down what was supposed to be the “Manhattan project for nutrition.”)
But after spending the past few days talking with some of the brightest minds in nutrition research and epidemiology, we now feel the PREDIMED retraction is actually cause for hope. Maybe even a new beginning for the field.
Yes, studies with big flaws pass peer review and make into high-impact journals, but the record can eventually be corrected because of skeptical researchers questioning things. It’s science working as it should, and the PREDIMED takedown is a wonderful example of that. This process should bring us a step closer to what really matters: informing people who want to know how to eat for a healthy life.
Why the PREDIMED trial was such a big deal
As reported, nutrition science has done a great job of finding ways to address diseases of nutrient deficiencies, like scurvy. But today, our greatest health problems relate to overeating. People are consuming too many calories and too much low-quality food, bringing on chronic diseases like cancer, obesity, diabetes, and cardiovascular disease.
These illnesses are much harder to get a handle on. They don’t appear overnight but develop over years. They’re not usually related to one cause; they’re caused by many lifestyle and genetic factors in concert. And fixing them isn’t just a question of adding an occasional orange to someone’s diet. It involves looking holistically at lifestyle behaviors, like diet and genetics, trying to tease out the risk factors that lead to illness.
The trouble is, most of what we know about nutrition’s effects on chronic disease comes from observational data. Researchers track what large numbers of people eat over time and then look at their rates of disease, trying to tease out relationships in the data. Do people who drink more red wine have lower rates of heart disease? Is meat associated with an early death?
Because these studies aren’t controlled like experiments, they can’t tell us whether one thing caused another thing to happen. Researchers try to use statistics to control for some of these “confounding factors,” but it’s impossible to catch all of them.
That’s why the randomized controlled trial (RCT) is considered the gold standard for evidence in health research. With these trials, scientists randomly assign their study participants to one of two groups (or sometimes more). One group gets a treatment or intervention; a control group gets a placebo. Again, they give participants these assignments in a random fashion, meaning the members of the study have an equal chance of being selected for each group. And if there’s a difference in health outcomes at the end of the study, it’s fair to say the intervention was the cause.
But there’s a catch with RCTs: It’s extremely difficult and expensive to run this kind of study for long enough and in numbers that are large enough to yield meaningful answers.
This is exactly why PREDIMED stood out. “It was a randomized control trial, it was long-term, and it had clinical outcomes on things that mattered,” said Ioannidis. “It was the prototype of the best that had been done.”
What PREDIMED found
The study was conducted in Spain and tracked more than 7,400 people at high risk for cardiovascular disease, recruited through 11 study sites.
The study participants were randomly assigned to one of three groups: getting advice about following a Mediterranean diet as well as free extra-virgin olive oil delivered to their home; getting advice about following a Mediterranean diet as well as free nuts to their doorstep; and, for the control group, getting advice about following a low-fat diet.
The main endpoints the researchers tracked were major cardiovascular events — strokes, heart attacks, death from cardiovascular causes. They stopped the trial early, after a median follow-up time of nearly five years, because the effects of the diet seemed to be so dramatic. The Mediterranean diet, when supplemented with lots of olive oil or nuts, could cut a person’s risk of cardiovascular disease by a third.
It’s rare to see an effect size that big, even in studies on medications. What’s more, the study’s early halting seemed to exaggerate the results in the minds of the public.
“The magnitude of the diet’s benefits startled experts,” the Times reported when the trial came out. “The study ended early, after almost five years, because the results were so clear it was considered unethical to continue.”
What went wrong with the study
The findings helped heave olive oil and nuts into the realm of the sacred. They were catnip for journalists (“Mediterranean Diet Shown to Ward Off Heart Attack and Stroke,” the New York Times story read. “Spanish Test: Mediterranean Diet Shines In Clinical Study,” proclaimed NPR’s headline.) And they spawned a cottage industry of studies by other scientists, who used the PREDIMED data to run hundreds of different analyses.
But it turns out the trial wasn’t properly run.
John Carlisle, a British anesthesiologist (and legend in medical statistics), first called attention to the fact that something seemed fishy about PREDIMED in this June 2017 paper. In the study, he applied a statistical test to 5,087 randomized trials, including PREDIMED, to see if the groups randomized in the studies were too similar to happen by chance. He found they were, suggesting problems with the trial’s randomization process.
One of PREDIMED’s authors, nutrition researcher Miguel Ángel Martínez González, told Vox that upon seeing the Carlisle paper, he reached out to his colleagues on PREDIMED’s steering committee about reexamining the data. He told me they also wrote to the NEJM about the potential errors, which led to a plan to recover old data and fish for errors — though the editor of the journal told Science it was they who reached out to the researchers.
Martínez González, his colleagues, and editors at NEJM spent months poring over the data. They discovered several major problems:
- The researchers at one of the study sites, overseeing 11 clinics, failed to randomly assign individual people to the different diets, instead assigning everybody in the same clinic to the same diet.
- In cases where more than one person in a family was participating in the study, they were all assigned to the same diet, again instead of being randomized. (And while the authors knew about this deviation, it wasn’t reported in their original study as it should have been.)
- Researchers at another study site failed to properly use the randomization table, which is supposed to guide researchers in how the randomization is done.
- Altogether, these errors meant that some 1,588 of the study’s participants — or a little more than 20 percent — weren’t properly randomized in one way or another.
Needless to say, this isn’t how a randomized trial should work. The researchers should have made sure each study site and clinic involved in the trial was following strict procedures on how to randomize, continually following up with and checking in on them, Indiana University School of Public Health dean David Allison told me.
Martínez González insists that the problems with the randomization were not clinically meaningful, that they don’t fundamentally change the conclusions of the study. “The aim of randomization is to get balanced groups. There were no imbalances in the three groups regarding the clinical characteristics,” he said.
Mainly for that reason, the researchers only slightly watered down their conclusions in the revised version of the study published on June 13. “We are now more sure than ever of the strength of PREDIMED and the robustness of the results,” Martínez González told me. “We honestly reported these small departures from the strict randomization procedure — but this has no importance regarding the methods and results of the trial.”
“We have seen that this is the deepest scrutiny ever done to a trial,” he added. “I think that if you put the magnifying glass to any published trial, things will appear.”
Other scientists remain skeptical of the retooled PREDIMED
Researchers who weren’t involved in the study aren’t so sure about PREDIMED’s retooled conclusions. They feel the randomization problems suggest sloppiness or deliberate data manipulation and that upon further scrutiny, more errors will indeed materialize that will shake PREDIMED’s conclusions even more.
“They only came to know what was wrong with the randomization after it was published? That’s really weird, and then I get nervous about [other] details,” said Cecile Janssens, a research professor of epidemiology at Emory University. “It was weird they retracted it and weird they republished it immediately, and I feel there may be other things wrong with this study.”
On her Twitter feed, she’s now pointing out flaws in the retooled report and asking others to help her do the same:
Hilda Bastian, who also specializes in picking apart trials, has pointed out other red flags in the rebooted trial on her blog. “The reboot, and the choice of data and calculations reported, leave a lot of questions,” Bastian told me over email. “It begs the question of how impartial a journal is able to be, when one of its star papers explodes.”
For Richard Bazinet, a University of Toronto nutrition researcher, the biggest worry isn’t just that the study itself is wrong; it’s all the other studies out there that are based on PREDIMED’s data.
“People have taken that [PREDIMED] data set and reanalyzed it for other things over and over and over again. There are probably a hundred papers in the literature. I can’t even figure out what the process is going to be to [fix this],” he said. “That’s the thing that’s got me up at night.”
PREDIMED should be a beginning, not an ending
We can probably expect to hear more about PREDIMED in the coming months. And the only way to move toward better and more robust nutrition science is to learn from the debacle.
“A big [lesson] is how important it is to take post-publication critiques seriously and act on them,” said Bastian.
While many people had pointed to flaws in the trial over the years, it was Carlisle’s statistical critique, and the fact that it wasn’t ignored, that led to the retraction. “The better, critical systematic reviews had already pegged this study as being particularly weak evidence, and they needed to be taken more seriously too,” she added.
Another big issue is the need for improved transparency, Bastian said. “For such a big trial, the protocol, methodological detail, and even data are extraordinarily skimpy in key areas — and that goes for the reanalysis too.”
“I wouldn’t give up [on nutrition science],” said Ioannidis. “But we need to take at a minimum some steps forward: Perform more, long-term, large randomized control trials — like PREDIMED but do it right. We need to share the data, and make them publicly available, have people be able to look at them and see that they get the same results.” People in many other fields of science, notably psychology, are calling for similar steps to strengthen science.
For now, he added, “[Nutrition epidemiology] is a field that’s grown old and died. At some point, we need to bury the corpse and move on to a more open, transparent sharing and controlled experimental way.” PREDIMED is probably that point.
So what should I eat?
The PREDIMED retraction isn’t a total dismissal of Mediterranean eating patterns, but it’s a reminder of just how difficult it is to come by clear advice on precisely what people should eat for their best health.
Layered on top of the difficulties with studying nutrition is the fact that people have very strong feelings about food — from scientists to study authors to the media — and these feelings bias the research and how it’s interpreted. Food is cultural, it’s social, it’s about our family histories and where we grew up, and it’s something we all have experience with, Allison noted. It’s no wonder “people become zealots” when they talk about diet.
Despite what the zealots say, there’s lots of data suggesting humans can survive — and even thrive — on myriad eating patterns. The optimal diet for any individual probably varies, but in general, you can’t go wrong with lots of fruits and vegetables, little junk food, and not too many calories. If you don’t smoke too much, don’t drink too much, and exercise regularly, you’ll probably be okay. Of courses, that’s easier said than done for the vast majority of people.
