Why Brain Imaging Doesn’t Always Hold Up—and How Experts Say to Fix It

Neuroimaging is facing a reproducibility crisis, Yale School of Medicine (YSM) researchers say.

One way scientists use neuroimaging is to create maps of the brain and try to explain how differences in structure and function relate to differences in a person’s behavior. However, these maps don’t always hold up when applied to new data.

Now, in a Nature Methods article, YSM researchers call for the use of external validation in neuroimaging research to improve its replicability. External validation involves taking a model created for one dataset and testing it in a completely independent dataset. Its use, they argue, not only leads to more robust science, but also can help researchers catch mistakes faster.

“If we want the field to be replicable and reproducible, we need to use external validation,” says Matthew Rosenblatt, PhD, a former graduate student at YSM and the paper’s first author.

The goal of neuroimaging research is to uncover biology that spans across groups of people. “If you’re mapping a relationship between the brain and behavior, you don’t want it just to be super specific to the several hundred participants you’re looking at in a very specific dataset,” Rosenblatt says.

Generalizability describes how well brain mapping applies to independent data. Maximizing generalizability is the ultimate goal for scientists developing these models, Rosenblatt explains.

Poor generalizability is especially a problem in neuroimaging research. This is due, in part, to the use of smaller sample sizes as collecting neuroimaging data is time consuming and costly. “We don’t have the same advantages of fields like AI and computer science or genetics where they have hundreds of thousands of participants,” Rosenblatt says.

AI models tend to work better for some groups of people than for others. The most common theory for why this bias exists is that certain demographic characteristics tend to be overrepresented or underrepresented in datasets. Certain demographics—including participants who are from Western countries, a high socioeconomic background, and are highly educated—tend to be overrepresented, for example.

To validate findings, scientists might take the model they built using one dataset and apply it to a different dataset with an entirely new cohort of participants.

The independent dataset might also have slightly different characteristics than the original. “Datasets do not need to perfectly match to still be useable,” Rosenblatt says. “Our goal is to map brain-to-broad constructs, so it’s not prohibitive if datasets are slightly different.”

There are several different ways, for example, to measure depression symptoms, including the Beck Depression Inventory, the Montgomery-Åsberg Depression Rating Scale, and the Hamilton Depression Rating Scale. “Those all have slightly different questions and slightly different properties,” says Dustin Scheinost, PhD, associate professor of radiology and biomedical imaging, associate director of biomedical imaging technology at the Yale Biomedical Imaging Institute, and the paper’s senior author. “But if we establish a robust brain-behavior association, the model should work across all of them.”

The use of external validation can improve the generalizability of neuroimaging models, the researchers argue. “If you show that your model extends beyond these participant and dataset differences, it’s likely more robust,” Rosenblatt says.

External validation can also help catch mistakes in the data such as data leakage, which occurs when scientists accidentally introduce information from outside the training dataset. This can falsely inflate the model’s performance. “Unfortunately, some of these mistakes get published,” Scheinost says. “The quicker you can catch them, the quicker it stops people from going down the wrong line of research.”

External validation also protects against data manipulation. “This is unfortunately a possibility,” Rosenblatt says. “But that model would not generalize to an external unmanipulated dataset.”

The biggest mistake scientists make when conducting neuroimaging research, says Scheinost, is not doing external validation at all. “Scientists are often trained that everything should be fixed and highly precise,” he says. “They tend to get scared that their datasets are too different.”

If a model fails external validation, it could indeed be a sign that the independent dataset was too different. But failure could also point to scientifically interesting insights. For example, a model might work for a cohort of men, but not women, signaling possible sex differences in this particular aspect of the brain. Or a model might work for an older cohort, but not a younger one.

“Maybe a model that predicts cognition in adults doesn’t work in kids,” Scheinost says. “That could tell you something about development.”

In recent years, the number of publicly available neuroimaging data sets has grown dramatically. Sharing data is one of the most important things scientists can do to promote the use of external validation, the researchers say, because it offers others more datasets to pick from.

The growing use of external validation will promote robust brain mapping studies that could better enable scientists to translate findings on the brain and behavior into practical interventions, the researchers say.

“We want science to be robust and rigorous,” Scheinost says. “In terms of external validation, my advice is to do it, don’t be scared to try it, and don’t take its failure as a failure of the model."