The cross-site reproducibility of MRI morphometric phenotypes in psychiatric disorders
Abstract: Decades of structural magnetic resonance imaging (MRI) studies have documented alterations of gray matter morphometry in psychiatric disorders, but the field has failed to identify any consensus disease phenotypes. Here we examine whether current approaches will ever converge on such phenotypes by evaluating the consistency of brain-wide maps of gray matter volume and cortical thickness differences obtained for each of 59 study sites of five psychiatric disorders (schizophrenia, schizoaffective disorder, autism spectrum disorder, major depressive disorder and bipolar disorder), totaling 2,437 patients and 2,065 controls.
We find that cross-site consistency is low (median r ≤ 0.16); markedly reduced compared to Alzheimer’s disease (r = 0.54); unexplained by demographic, clinical or scanner differences; and robust to analytic choices. Using bootstrapping, we observe that consistency may improve for sample sizes ≥200 per group for schizophrenia but that other disorders may require much larger samples. Our findings indicate that current widespread practices in structural MRI are unlikely to identify robust morphometric phenotypes for psychiatric disorders.
Commentary: The researchers gathered structural MRI scans from thousands of people across dozens of study sites, covering five psychiatric diagnoses, and processed all of the scans using the same methods. At each site, they compared people carrying a diagnosis with control participants and created a map showing where the two groups appeared to differ in cortical thickness or gray-matter volume. They then compared those maps across sites to see whether independent studies were finding the same anatomical pattern. For the psychiatric diagnoses, the maps were generally inconsistent and often bore almost no resemblance to one another, unlike the much more consistent pattern found for Alzheimer’s disease. The researchers then pooled participants from all sites and repeatedly divided them into two large artificial groups, finding that the resulting maps became more similar as the groups grew, particularly for schizophrenia.
The holy grail of cognitive science and psychiatry has always been to demonstrate that behavioral descriptions such as “attention” or “autism” correspond to stable, independently recoverable physiological organization. That is the threshold these fields repeatedly promise will carry their concepts from folklore into hard science. Despite decades of work, they have largely failed to cross it, and the evidence increasingly suggests that many of these concepts cannot be rescued in their present form.
The article is an excellent piece of work that demonstrates exactly how little physiological validity these concepts have acquired. Yet at the moment when its results could deliver a decisive blow against the folklore, it retreats into rescue mode. Concluding that decades of psychiatric imaging have been organized around categories that do not identify coherent biological entities would threaten the premise supporting the field itself. So the failure is recast as a demand for larger samples, better subgroups, and more sophisticated analysis. The paper then commits the same validity error it has just exposed, treating a stable average produced by pooling non-equivalent populations as evidence that a genuine “disease phenotype” was hiding underneath. More data can stabilize an artifact of classification without validating the classification.
The appeal to more data is especially hollow because this is not a young field waiting for its first mature datasets. We have spent decades diagnosing, treating, imaging, and following these populations across development, crisis, recovery, relapse, medication exposure, and aging. If these categories identified stable physiological entities, that enormous longitudinal record should have progressively sharpened their biological boundaries. Instead, larger datasets have mostly made small population averages more statistically stable while individual and cross-site coherence remain absent. The missing ingredient is not sample size. It is a valid target. Another generation of pooling cannot manufacture biological unity from categories that never demonstrated it.