Key takeaways

• Swapping the software that outlines brain regions barely changes the outlines themselves, but it measurably changes the numbers computed from them, and that reaches all the way through to prediction accuracy (77-84%).
• In small, tightly controlled datasets, hand-designed radiomic features and classical machine learning still beat deep learning end-to-end: you get explainability you can’t get otherwise, and you don’t have enough data to feed a bigger model anyway.
• Fully automatic, deep-learning-based segmentation was the surprise winner here, producing the most robust and most predictive features of the four methods tested.

Radiomics

The word radi-omics is a combination of ‘radiology’ and the suffix ‘-omics’. Radiology is the main medical specialty to use medical imaging. It relies heavily on the interpretation of images by medical experts. This is called the “qualitative” assessment. In contrast, the suffix “-omics” implies a measured approach: a “quantitative” approach. Radiology is the backbone of diagnosing diseases and guiding their treatment using imaging. Radiomics has proven useful alongside it in several applications, primarily in the diagnosis and treatment of cancer.

Applications

Scientific studies have found that radiomics can be useful in understanding conditions like schizophrenia, ADHD, bipolar disorder, and depression. Radiomics is good at spotting small changes in brain scans that the naked eye can’t see. But because it’s so sensitive, even small differences in how we measure brain properties can affect the results. This makes it hard for these applications to be used in hospitals. Radiomics’ sensitivity is both a blessing and a curse.

One source of issues for radiomics is the disagreement about how the brain regions are delineated. The delineation of brain regions is called segmentation. The segmentation is often performed using software. But each software tool creates (slightly) different segmentations. For example, an animation of the segmentations created by three different tools is shown below.

As with people, each delineation tool will produce slightly different delineations of brain regions.

Why hand-designed features, in the age of deep learning?

It’s a fair question: why extract manually engineered radiomic features at all, when deep learning can often learn features directly from an image? In this kind of work the honest answer is dataset size. Our cohort was 46 people, nowhere near enough to train an end-to-end deep network without overfitting, but plenty for classical machine learning on a set of well-chosen, explicit features. The other reason is explainability. In psychiatry and neuroscience, where we don’t have a lot of control over confounders in the data, being able to point at which measurement drove a prediction matters more than squeezing out an extra percent of accuracy. Radiomics gives you that; a black-box deep network mostly doesn’t.

Q&A

Segmentation is one part of several steps that take an image of the brain to statistical analysis. Commonly, segmentation stands at the middle of these steps. Segmentation is first followed by the extraction of measurements from the segmented regions. The extraction is then followed by the statistical analysis, or prediction using these measures. Our paper took a closer look at the disagreement among segmentation tools, and how this affected the radiomics analysis down the line.

Segmentation: Which brain parts are hardest to agree on?

Across the four segmentation methods we tested (two classical: FreeSurfer ASEG and SAMSEG, two deep-learning-based: FastSurfer and Med-DeepBrain), agreement was high everywhere, with a Dice-coefficient interquartile range of 0.77-0.90. But it wasn’t uniform. Smaller brain regions, which have relatively more surface area per unit of volume, suffer most from disagreement among segmentation tools: the pallidum was the clearest example, with a 0.1-point gap in Dice score between the left and right side that has no anatomical reason to exist. In the first figure of the published work, we show this agreement broken down by region. Researchers targeting these poorly-agreeing regions specifically should expect their findings to be harder to reproduce across labs using different software.

Extraction: Which measures are most dependable?

Once we have defined a brain region there are many things we can measure about it. Its color, its contrast, all kinds of properties about its shape and even patterns within the regions. There are so many that it is easiest to discuss them as seven groups, roughly in order of complexity: shape, image intensity (also known as first-order), and five groups of image intensity patterns. We found that measures with higher complexity were more affected both by 1) disagreement in segmentation tools, and by 2) scan-rescanning (scanning the same patient twice). Robustness to swapping segmentation methods was actually higher (ICC over 0.7) than reproducibility across a simple scan-rescan of the same person (ICC between 0.3 and 0.8): which software draws the outline matters less than we feared, at least compared to the noise already baked into scanning the same person on two different days.

Prediction: Does it matter in the end?

Finally, we tested if these changes affected how useful these measures were for predicting something. Interestingly, the data set we used was used before to test for changes in the brain after staying up all night. Healthy volunteers stayed at the hospital all night and were deprived of sleep. A control group went to sleep as normal and all were scanned before and after the night. This previous work already showed significant short-term changes due to sleep deprivation, which is interesting in its own right.

In this work, we used the prediction of sleep deprivation as a test case (46 subjects, split roughly 22 normal-sleep to 20 sleep-deprived). We trained a small neural network on the radiomic features produced by each of the four segmentation methods in turn, and compared how well each one classified sleep-deprived subjects:

Median classification accuracy per segmentation method, with interquartile range. FastSurfer, the newest deep-learning-based method, came out on top.

The two deep-learning-based methods outperformed the two classical ones, and FastSurfer’s edge over both ASEG and SAMSEG was statistically significant. That’s not a subtle effect: the same radiomics pipeline, run on the same scans, swings by up to 13 points of accuracy depending only on which tool drew the outlines.

Where I land now

The part of this project I’m most grateful for, in hindsight, is that fully automatic segmentation existed at all. A few years earlier, a study like this would have meant a research assistant manually tracing brain structures on 46 people, four times each. That’s not research, that’s a part-time job. Automatic, and increasingly deep-learning-based, segmentation is what makes work like this feasible at all.

At the same time, doing this work made me more skeptical, not less, about radiomics as a long-term approach for psychiatric and psychological applications specifically. Watching where deep learning and dataset sizes are heading, I don’t think the future of this field is a better hand-designed feature or a cleverer classical model. I think it’s more data. Once someone has a large enough psychiatric imaging dataset, I’d expect an end-to-end deep learning model to beat anything built on radiomics, the same way it already has in a lot of other imaging domains. Until then, in small and tightly controlled samples, radiomics and classical machine learning still earn their keep on explainability alone.