Convolutional Neural Network based frameworks for fast automatic segmentation of thalamic nuclei from native and synthesized contrast structural MRI

CNN Architecture Overview

Figure 1 shows an overview of the proposed thalamic nuclei segmentation frameworks. Fig. 1a shows the 3D Native Contrast Segmentation (NCS) CNN that predicts the whole thalamus and individual thalamic nuclei on MPRAGE images. Fig. 1b shows the Synthesized Contrast Segmentation (SCS) CNN where a 3D contrast synthesis CNN to generate synthesized WMn-MPRAGE representation from MPRAGE images is introduced prior to the segmentation CNN.

The multi-scale thalamic nuclei segmentation CNN (Fig. 2a) has a U-NET (Ronneberger, Fischer, and Brox 2015) like encoder-decoder framework that processes features at multiple scales. At each resolution level, a series of convolution (3x3x3), batch normalization, and rectified linear unit layers extract features followed by a pooling layer for downsampling. The skip connections from each resolution level are concatenated to the corresponding resolution level along the upsampling path. The segmentation CNN uses MPRAGE images as input and predicts both the thalamus and individual thalamic nuclei in one pass through the model. Multiresolution feature maps are first used to predict the thalamus label. These feature maps, concatenated with the predicted thalamus label from the previous step, are then used to localize individual thalamic nuclei. Whole thalamus and thalamic nuclei labels generated from WMn-MPRAGE images using THOMAS are used as surrogate for manual segmentation ground truth for training the segmentation CNN. Dice loss is used for the thalamus and multi-label Dice loss is used for thalamic nuclei. The total segmentation loss is the sum of errors in the prediction of the thalamus using Dice loss and its individual nuclei using multi-label Dice loss. If C refers to the total number of nuclei labels, G_i and P_i refer to the ground truth annotations and predictions for the i^th label, respectively, the multi-label Dice loss used in this work is defined as follows:

The architecture for the multi-scale contrast synthesis CNN also uses an encoder-decoder architecture similar to the segmentation CNN (Fig. 2b) except for the loss formulation. The contrast synthesis CNN is trained using MPRAGE images as input and co-registered WMn-MPRAGE images as ground truth. Here, along with an intensity similarity-based loss function, we also use the perceptual loss measured using a VGG16 model (Simonyan and Zisserman 2015) pre-trained on ImageNet, similar to some other works on contrast synthesis in MR images (Dar et al. 2019). If W and W_syn are the ground truth WMn-MPRAGE image and the WMn-MPRAGE image synthesized from the corresponding MPRAGE image, respectively, the total synthesis loss function is given as:

The first term represents the intensity similarity-based loss defined as the L₁ norm-error between the ground truth and the synthesized images. The second term, corresponding to the L₂ norm error of the perceptual loss, ensures visual similarity of the synthesized images and the reference images. The function F(.) extracts feature maps from the third rectified linear unit layer of the ImageNet pre-trained VGG16 model to compare features from ground truth and synthesized images.

Study Cohort

For generalizability, images acquired on two different scanners were pooled for training. This comprised of data from 18 healthy individuals acquired on a 3T MAGNETOM Prisma (Siemens Medical Solutions USA, Inc., Malvern, PA, USA) and 17 healthy individuals acquired on a 3T Signa (General Electric Healthcare, Waukesha, WI) MR scanner, each subject being scanned with both MPRAGE and WMn-MPRAGE protocols. Of these 35 individuals, 33 subjects were retained for training, 2 were used for validation. Note that the training and validation cohorts consisted entirely of healthy individuals with no known pathologies.

Two independent test cohorts were used to demonstrate generalizability. The performance of the thalamic nuclei segmentation CNN trained directly on MPRAGE images (i.e. NCS) with those using synthesized WMn-MPRAGE images (i.e. SCS) is first evaluated on a cohort of healthy subjects (n=12). This cohort is referred to as test cohort 1. For clinical validation, the CNNs proposed in this work are used on MPRAGE images from an independent cohort of subjects (n=45) with alcohol use disorder (AUD) (n=22) and gender- and age-matched controls (n=23) to study the effect of AUD on thalamic nuclear atrophy. This second test cohort is referred to as the AUD cohort. The WMn-MPRAGE images from this cohort were previously used to characterize atrophy in thalamic nuclei volumes in alcoholism using THOMAS (Zahr et al. 2020). The image acquisition for the testing cohorts also consisted of MPRAGE and WMn-MPRAGE protocols. All imaging data was acquired after prior informed consent and adhering to institutional review board guidelines. Additional information on MR image acquisition protocols for the subjects used in this study are available in the Supplementary Table 1.

CNN Data Generation

The ground truth labels for the thalamus and 11 thalamic nuclei were generated on WMn-MPRAGE images using THOMAS (Su et al. 2019), a multi-atlas segmentation technique. THOMAS uses 20 anatomical priors with manual delineations by a radiologist, guided by the Morel atlas, to segment the thalamus into 11 nuclei and the mammillothalamic tract in 15 minutes or less.

The isotropic resolution MPRAGE images and the corresponding WMn-MPRAGE images were co-registered using an affine transformation to train the segmentation CNNs. The co-registered images were brain extracted using FSL’s brain extraction toolbox (Smith 2002), N4-bias corrected (Tustison et al. 2010), and intensity normalized using histogram-based contrast stretching to scale intensities in the range of [0, 1]. Contrast stretching was performed to clip intensities below the 1^st percentile and above the 99^th percentile to avoid outlier-based clipping that may affect traditional min-max based intensity normalization.

The segmentation CNNs were trained using an image-based scheme. From each subject, training data for the segmentation CNN were generated by cropping and extracting 2.5D (192x192x5) images using a sliding window. Data augmentation for the segmentation CNN training included scaling and shearing transformations. The contrast synthesis CNN was trained using a patch-based scheme. The co-registered image pairs (MPRAGE and WMn-MPRAGE images) were used to extract overlapping 2.5D patches (64x64x5) using a sliding window over the entire brain. The patches were extracted only from regions within the brain mask for each subject. Data augmentation for the synthesis CNN included random in-plane and through-plane rotations in addition to scaling and shearing transformations. Augmented images were generated on every training image batch by randomly selecting a subset of data augmentation schemes described previously during run time.

CNN Experiments and Implementation

To understand how segmentation CNN performance changes with loss function, we also trained the NCS CNN with weighted categorical cross entropy loss (WCCE). Here, a binary cross-entropy loss was used for thalamus and WCCE was used for thalamic nuclei. The WCCE loss function is defined as:

Here, C is the number of labels, w_i is the weight associated with the i^th label, g_n and p_n refer to the n^th pixel of the ground truth (G) and prediction (P), respectively.

The training parameters for the thalamic nuclei segmentation CNN are: loss function for thalamus = Dice, loss function for thalamic nuclei = multi-label Dice, epochs = 50, batch size = 10, optimizer = Adam, learning rate = 0.001, and decay = 0.1. The training parameters for the contrast synthesis CNN are: Loss function for image similarity = L1, perceptual loss = L2, epochs = 50, batch size = 10, optimizer = Adam, learning rate = 0.001, decay = 0.1. Experiments were implemented in Python using Keras (https://keras.io) with Tensorflow (https://www.tensorflow.org) backend on a Linux system, with Titan P100 (NVIDIA) GPU. Architecture details and code used in this work are available at the following URL: https://github.com/lunastra26/thalamic-nuclei-segmentation.

Evaluation of CNN Performance

The whole thalamus and individual thalamic nuclei labels were predicted on MPRAGE images using the two segmentation frameworks. Performance of the segmentation CNN was evaluated using Dice score and percent volume difference (VD). If G and P refer to the ground truth THOMAS labels and predictions, respectively, then, $\text{Dice}=\frac{2\mid G\cap P\mid }{\mid G\mid +\mid P\mid }$ and $\text{VD}(\%)=\frac{\mid V_G-V_P\mid }{V_G}\times 100$. Here. V_G corresponds to the volume of mask G. The performance of the contrast synthesis CNN was evaluated using root mean squared error (RMSE), structural similarity (SSIM), and peak signal-to-noise (PSNR) ratio. The definitions for these evaluation metrics are presented in the Supplementary material.

Statistical Analysis

A paired two-sided t-test (α = 0.05) was used to compare if there were any significant differences in the performance of the segmentation CNNs trained on MPRAGE images compared to their WMn-MPRAGE representation. A Bland Altman analysis was performed to assess agreement between ground truth and volumes predicted by NCS and SCS CNNs. Pearson’s correlation coefficient, coefficient of variation (CV) and percent reproducibility coefficient (RPC) statistics were also computed.

A recent work by Zahr et al (2020) used WMn-MPRAGE images and THOMAS segmentation to detect significant atrophy in the ventral lateral posterior (VLp) nucleus on a cohort comprising of patients with alcohol use disorder (AUD) and age-matched controls. We used the AUD cohort from this work to characterize atrophy in thalamic nuclei volumes in alcoholism, but using MPRAGE images instead of WMn-MPRAGE. The 11 thalamic nuclei were combined into four groups similar to Zahr et al (2020) : anterior, lateral, posterior, and medial. The nuclei breakdown for these groups is shown in Table 1. ANCOVA analysis was first performed on the predicted volumes from the NCS and SCS CNNs to understand the relationship between volumes of the nuclei groups and diagnosis (controls vs alcoholics), after accounting for age and intercranial volume (ICV) estimated using Freesurfer segmentation (Buckner et al. 2004). If a significant relationship was observed within any of these groups, ANCOVA analysis was repeated for individual nuclei within that group to further understand relationship between thalamic nuclei volumes and diagnosis.

Synthesis CNN Performance

Synthesized WMn-MPRAGE images generated from the contrast synthesis CNN are compared to input MPRAGE and target WMn-MPRAGE images in Figure 3. The contrast synthesis CNN is able to mimic native WMn-MPRAGE contrast, especially between thalamus and white matter and inside the thalamus. A quantitative evaluation of synthesis CNN (Table 3) shows that synthesized WMn-MPRAGE images have a high structural similarity (93.7% and 92.1%) and PSNR (23.5 and 21.3) on both test cohorts. Joint histograms between co-registered MPRAGE, WMn-MPRAGE, and synthesized WMn-MPRAGE images for the thalamus region from two test images (Supplementary Figure 1) show high concordance between histograms of WMn-MPRAGE images and synthesized WMn-MPRAGE images.

Segmentation CNN Performance

A qualitative comparison of the thalamic nuclei segmentation predictions from NCS and SCS CNNs for a test subject are shown in Figure 4. Note the improved intra-thalamic contrast as well as thalamus-WM boundary in SCS (arrows) compared to NCS. Both the CNN models are able to accurately identify thalamic nuclei relative to the ground truth for the larger nuclei. However, the boundaries of the smaller nuclei such as VA, Hb, and MTT (Figure 4) and AV, VLa, CM, LGN, MGN (Figure 5) from SCS CNN agree better with the ground truth annotations.

On the AUD cohort (Table 4), the segmentation CNNs yielded Dice scores of at least 0.84 for the larger nuclei (Pul, MD, VLp) and at least 0.7 for the smaller nuclei. We also observed that synthesizing WMn-MPRAGE representation prior to segmentation significantly reduced overall VD of the whole thalamus (P=.008). For nuclei such as MGN (P=.003) and CM (P=.01), the predictions from SCS CNN yielded a significantly higher Dice score (MGN, P=.003 and CM, P=.01). Although the Dice scores were relatively comparable, SCS CNN made smaller errors in volume estimates compared to NCS in most of the nuclei, with significant lower error in volume estimates for VPl (P=.01) and VA (P=.001). Similar trends were also observed in test cohort 1 (Supplementary Table 2) where segmentation using synthesized WMn-MPRAGE representation instead of MPRAGE images yielded significantly improved Dice scores (AV and CM, P<.005) and VD (CM, P<.005; MD, P<.01) for some nuclei and comparable performance in others.

Effect of Diagnosis on Thalamic Nuclei Volumes in AUD Cohort

The least-squares mean (and standard error) from the ANCOVA analysis on the AUD cohort are presented in Table 5. There was a significant effect of diagnosis on lateral nuclei group volumes after controlling for age and ICV. This reduction in lateral group of nuclei volumes in alcoholics was observed in volumes obtained from both SCS (F [1, 39] = 6.96, p = 0.01) and NCS CNNs (F [1, 39] = 6.92, p = 0.01). No significant effect of diagnosis was observed on other nuclei groups (anterior, medial, or posterior). Among the constituent nuclei of the lateral group (VLp, VPl, VA, VLa), both SCS (F [1, 39] = 6.68, p = 0.01) and NCS CNNs (F [1, 39] = 5.96, p = 0.04) segmentations showed a significant atrophy in VLp nucleus volumes in alcoholism. Note that these results are consistent with the previously reported study by Zahr et al (2020) where a significant atrophy was observed in the VLp nucleus in alcoholics but using WMn-MPRAGE images segmented using THOMAS. In addition to VLp, we noticed that the NCS CNN also showed a significant reduction in VPl (F [1, 39] = 7.92, p = 0.008) nucleus for subjects with AUD. However, neither Zahr et al (2020) nor the SCS CNN framework showed any significant atrophy in the VPl nucleus.

Bland-Altman plots to assess agreement in CNN-predicted volumes and the ground truth for the whole thalamus, VLp, and the VPl nuclei for the AUD cohort are shown in Figure 6. A stronger correlation between ground truth and volume estimates from SCS CNN is observed compared to NCS CNN. The limits of agreement were also tighter when using synthesized WMn-MPRAGE images for the whole thalamus (Fig. 6d) and VPl nucleus (Fig. 6f) compared to using MPRAGE images (Fig. 6a and Fig. 6c). Bland-Altman plots for NCS CNN show a higher CV for the thalamus (7% vs 4% from SCS CNN) and VPl nucleus (17% vs 10% from SCS CNN).

Figure 7 (and Supplementary Table 3) also compares Dice scores and VD from the NCS CNN trained on WMn-MPRAGE contrast to demonstrate the improvements in segmentation accuracy if true WMn-MPRAGE images were used for training the segmentation CNN. We see that although synthesized WMn-MPRAGE images offer an improvement over MPRAGE contrast, true WMn-MPRAGE contrast still yielded better performance in terms of improved Dice scores and reduced VD for all thalamic nuclei (Figure 7). However, in scenarios where acquisition of WMn-MPRAGE is not possible, such as existing clinical protocols and online repositories, the use of synthesized WMn-MPRAGE can provide improvements over MPRAGE based thalamic nuclei segmentation.

WMn-MPRAGE Contrast as an Efficient Representation for Thalamic Nuclei Segmentation

Although both the segmentation CNNs showed comparable performance on the thalamus and larger thalamic nuclei on the AUD cohort, the use of SCS CNN was able to significantly improve the Dice scores (MGN and CM) and volume difference (VA and VPl) for some nuclei compared to NCS CNN, and replicate previously observed thalamic nuclear atrophy trends in AUD patients. Both the NCS and the SCS CNNs were able to replicate atrophy in VLp nuclei for subjects with AUD (Zahr et al. 2020) but the NCS CNN showed an additional spurious atrophy in VPl volumes. Note that this is consistent with the significantly higher VD in Table 4 (13.3% ± 12.7% in NCS vs 8.1% ± 6.2% in SCS, P<.008) and larger standard error in Table 5 for in VPl nucleus. The Bland-Altman plots (Figure 6) reveal that NCS under-estimated volumes from subjects with AUD, artificially inflating the group differences in subjects with thalamic nuclei atrophy due to the underlying pathology. Together, these explain the spurious atrophy in VPl nucleus volumes observed in alcoholics when using volumes computed from the MPRAGE images (i.e. NCS). By transforming the MPRAGE images to a new representation for the segmentation CNN, we were able to demonstrate atrophy only in the VLp nucleus, as previously reported by Zahr et al (2020) using THOMAS and true WMn-MPRAGE data, attesting to the sensitivity, accuracy, and potential utility of representational learning in the segmentation framework used in this work.

Inference Time Computational Efficiency and Existing literature

Manual annotations for the thalamus and thalamic nuclei on structural MR images, although considered gold-standard, are tedious, often requiring hours for accurate delineation. Automated thalamic nuclear segmentation techniques on MRI provide a promising alternative to time-consuming manual annotations. Although recent techniques such as the Bayesian probabilistic atlas-based thalamus segmentation (Iglesias et al. 2018) available in Freesurfer have been proposed for MPRAGE images, the processing time for Freesurfer-based segmentation is on the order of several hours. The shape-based multi-atlas technique proposed by Liu et al (2020) does not take advantage of inter-thalamic nuclear contrast and the segmentation accuracy can be affected in the presence of thalamic nuclear atrophy due to neurological diseases. Moreover, these existing atlas-based segmentation techniques on structural MR images (Iglesias et al. 2018; Su et al. 2019; Liu et al. 2020) rely on success of the nonlinear registration process to accurately transfer thalamic nuclei labels from atlas space to native space. In addition to being computationally burdensome with longer processing times, the accuracy of the segmentation is often limited by the success of the registration process. The CNN-based segmentation frameworks proposed in this work provide a fully automated, fast, and reliable technique for thalamic parcellations (11 nuclei and MTT) that neither requires registration to a template during inference time nor cropping of the thalamus region to predict individual thalamic nuclei. With prediction times in the order of seconds for a 3D volume, this is a significant reduction compared to multi-atlas schemes like THOMAS (15 min) or probabilistic atlas-based schemes like Freesurfer (several hours).

Recently, a multi-planar 2D CNN-based method was proposed for thalamic nuclei segmentation on WMn-MPRAGE images, where Majdi et al (2020) demonstrated preliminary results of thalamic nuclei segmentation from MPRAGE images using training from WMn-MPRAGE and MPRAGE contrasts. A cascade of 2D CNNs first predicted the thalamus and subsequently the individual thalamic nuclei. The input to the CNNs were cropped images obtained by registering them to a bounding box for thalamus on a WMn-MPRAGE template. This 2D CNN framework was shown to significantly outperform Freesurfer-based segmentation in terms of Dice scores, where the Freesurfer-based segmentation yielded significantly lower Dice scores for small nuclei such as AV, VLa, CM as well as LGN and MGN (Majdi et al. 2020). As opposed to a cascade of 2D CNNs in Majdi et al (2020), our proposed segmentation framework uses a single 3D CNN to directly predict the thalamus and individual nuclei on MPRAGE images in one pass through the CNN. Furthermore, NCS framework does not require any time-consuming registration-based preprocessing step for cropping.

Limitations and Future Work

A limitation of our work is the lack of manual annotations to train the segmentation CNNs. Manual annotations by experts, although considered gold standard, can be impractical to obtain for a large cohort such as ours. In this work, THOMAS, a multi-atlas framework that uses non-linear registration on WMn-MPRAGE images, is used to generate ground truth labels for training the CNN. The choice of target labels for the thalamus and individual thalamic nuclei, especially one generated from another automated segmentation technique, may affect the segmentation accuracy. However, THOMAS generated nuclei labels have previously been validated against manual annotations (Su et al. 2019). Evaluated on healthy volunteers and patients with Multiple Sclerosis by comparing against manual annotations, THOMAS achieved high Dice scores of at least 0.85 for the larger nuclei and at least 0.7 for smaller structures. Su et al (2019) also compared structural-MRI based THOMAS and Freesurfer-based thalamic nuclei segmentation to manual annotations on a small subset of cases and found that THOMAS outperformed Freesurfer in terms of Dice score for smaller nuclei such as CM, AV, and VA as well as in larger nuclei such as VLp and Pul. With improved performance against benchmarks, THOMAS-generated labels act as excellent surrogates for time-consuming manual annotations.

Although the segmentation CNNs proposed in this work yield registration-free predictions during inference time, the impact of the success of registration process in the THOMAS label-generation process on CNN predictions would need to be further investigated. Future studies may have to compare THOMAS and CNN generated labels with manual annotations, if and when available, to understand how errors propagate when training on labels generated by an automated algorithm instead of manual contours. The overall segmentation accuracy of the frameworks can be affected and further improved by the choice of CNN architectures, such as the use of CNN ensembles (Li et al. 2018; Umapathy et al. 2021) for improved generalization performance, or generative adversarial networks (Yi, Walia, and Babyn 2019) for contrast synthesis.

In this study, the segmentation frameworks were trained on a normal cohort and evaluated in subjects with AUD. Improvements in segmentation performance of the CNN was observed when training on a WMn-MPRAGE representation of the MPRAGE images, leading to improved sensitivity to the effects of AUD on thalamic nuclear atrophy on this particular cohort. The clinical applicability of the proposed technique to other neurological diseases that differentially affect thalamic nuclei volumes as well in the presence of lesions or metal electrode artifacts (for e.g. from deep brain stimulation leads) needs further investigation. Online repositories such as ADNI or OASIS with MPRAGE images can be used to explore the relationship between thalamic nuclei volumes and cognition. A fast and reliable MPRAGE-based thalamic parcellation technique such as ours would allow retrospective analysis of these previously acquired neuroimaging data.