Predicting progression from normal cognition to mild cognitive impairment for individuals at 5 years

Study design

The BIOCARD study, the parent study from which these data are drawn, was initiated at the National Institutes of Health (NIH) in 1995. While at the NIH, subjects were administered a neuropsychological battery and clinical assessments annually. MRI scans, CSF, and blood specimens were obtained approximately every 2 years. The study was stopped in 2005 for administrative reasons and re-established at Johns Hopkins in 2009, at which point the annual clinical and neuropsychological assessments were reinitiated. Bi-annual collection of CSF and MRI scans was re-established in 2015, as well as the acquisition of PET scans using Pittsburgh Compound B (PiB) (see Supplementary Fig. 1 for a schematic representation of the study design).

Consensus diagnostic procedures

Clinical and cognitive assessments were completed annually at the NIH initially and subsequently at Johns Hopkins, as noted above. A consensus diagnosis for each study visit was established by the staff of the BIOCARD Clinical Core at Johns Hopkins (prospectively for subjects evaluated starting in 2009 and retrospectively for subjects evaluated at the NIH). As previously described (Albert et al., 2014), each consensus diagnosis was handled in a similar manner. First a syndromic diagnosis was established: (i) clinical data pertaining to the medical, neurologic and psychiatric status of the subject were examined; (ii) reports of changes in cognition by the subject and by a collateral source were reviewed; and (iii) decline in cognitive performance, based on review of longitudinal testing from multiple domains (and by comparison with published norms) was determined. If a subject was deemed to be impaired, the decision about the likely aetiology of the syndrome was based on the medical, neurological, and psychiatric information collected at each visit, as well as medical records obtained from the subject, where necessary. More than one aetiology could be endorsed for each subject (e.g. Alzheimer’s disease and vascular disease). The consensus diagnosis procedures followed the diagnostic recommendations incorporated in the NIA-AA working group reports for the diagnosis of MCI (Albert et al., 2011) and dementia due to Alzheimer’s disease (McKhann et al., 2011).

The estimated age of onset of clinical symptoms was established separately, based primarily on a semi-structured interview with the subject and the collateral source. The staff conducting the consensus diagnoses were blinded to the CSF and MRI measures and to the APOE status of the participants (see Supplementary material, section 2 for additional details regarding the diagnostic procedures).

Selection criteria for variables included in the analyses

The ROC analyses presented here include variables from the four primary domains evaluated in the BIOCARD study, obtained when subjects were first enrolled. These domains included: (i) CSF values; (ii) MRI measures; (iii) cognitive test scores; and (iv) APOE genetic status. To be as parsimonious as possible, we based the selection of which specific variables should be included in the ROC analyses on findings from prior publications in which we had conducted Cox regression analyses designed in a parallel fashion. In each of these prior publications we used Cox regression procedures to examine the relationship between the values obtained at baseline (when participants were cognitively normal) and time to onset of clinical symptoms consistent with a diagnosis of MCI due to Alzheimer’s disease. Since the measures in these prior analyses had been standardized (using z-scores), it was possible to not only determine the relationship between the baseline measure and the outcome of interest (onset of clinical symptoms) but to also directly compare the hazard ratios across variables and domains. The measures selected from these prior analyses are described below.

Cognitive assessments

The annual, comprehensive neuropsychological battery covered all major cognitive domains, including memory, executive function, language, visuospatial ability, attention, speed of processing and psychomotor speed (see Albert et al., 2014 for the complete battery). Of the 17 variables selected from the cognitive battery (based on exploratory plots of change patterns over time), nine were significantly associated with the outcome, i.e. time to onset of clinical symptoms (Albert et al., 2014). The majority of the significant associations pertained to tests of episodic memory. We selected the two cognitive measures with the strongest association between baseline and outcome, based on hazard ratios, to include in the ROC analyses: (i) Digit Symbol Substitution Test from the Wechsler Adult Intelligence Scale – Revised; and (ii) Verbal Paired Associates – Immediate recall from the Wechsler Memory Scale – Revised.

CSF assessments

The CSF specimens collected from the participants were analysed using the xMAP-based AlzBio3 kit (Innogenetics) run on the Bioplex 200 system. The assay procedures were identical to those used in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). CSF specimens were analysed in triplicate on the same plate. Three variables were generated from these analyses: (i) CSF amyloid-β; (ii) CSF total tau; and (iii) CSF p-tau (see Supplementary material, section 3 for further details of the CSF assays). We selected the two CSF measures that showed a significant association between baseline and time to onset of clinical symptoms, based on hazard ratios, to include in the ROC analyses: (i) CSF amyloid-β; and (ii) CSF p-tau (Moghekar et al., 2013). Although the ratios for CSF tau/amyloid-β and CSF p-tau/amyloid-β were also significantly associated with time to onset of clinical symptoms in our prior analyses (Moghekar et al., 2013), the use of the individual CSF measures allowed us to examine their incremental predictive value on an individual basis.

MRI assessments

The MRI scans acquired from the participants were obtained using a standard multi-modal protocol with a GE 1.5 T scanner. We used the coronal scans to reconstruct the volumes of the entorhinal cortex, the hippocampus and the amygdala, as well as the thickness of the entorhinal cortex. The coronal scans used an SPGR (spoiled gradient echo) sequence (repetition time = 24 ms, echo time = 2 ms, field of view = 256 × 256, thickness/gaP = 2.0/0.0 mm, flip angle = 20°, 124 slices). The scans were processed with a semi-automated method, using region of interest large deformation diffeomorphic metric mapping (ROI-LDDMM) techniques (Miller et al., 2013) (see Supplementary material, section 4 for further details of these methods). We selected the two MRI measures that showed a significant association between baseline and time to onset of clinical symptoms, based on hazard ratios, to include in the ROC analyses: (i) right entorhinal cortex thickness; and (ii) right hippocampal volume (normalized by intracranial cavity volume) (Soldan et al., 2015).

APOE genotype

APOE genotypes were determined by restriction endonuclease digestion of polymerase chain reaction amplified genomic DNA (performed by Athena Diagnostics). APOE4 carrier status was coded by an indicator variable, with E4 carriers coded as 1 if the subject had at least one E4 allele and non-carriers coded as 0.

Summary of variables included in the analyses

The variables included in the ROC analyses, based on the selection criteria described above, were therefore as follows: (i) the Digit Symbol Substitution test and Paired Associates Immediate Recall scores from the cognitive domain; (ii) CSF amyloid-β and CSF p-tau from the CSF domain; (iii) right hippocampal volume and right entorhinal cortex thickness from the MRI domain; and (iv) APOE4 status from the genetics domain. Aside from the variables described above, all of the ROC analyses always included demographic variables (age, education), since all prior analyses indicated that these variables have important modifying effects on time to onset of clinical symptoms.

Each of the variables described above were continuous variables (with the exception of APOE status, which was a binary measure). All continuous measures were standardized (using z-scores) prior to inclusion in the ROC analyses. This makes it possible to directly compare the hazard ratios from each variable to one another (note that the hazard ratio for APOE is therefore not comparable to the hazard ratios for the continuous variables).

Table 2 presents the means and standard deviations of the measures in the analysis for subjects who remained normal over time versus those who progressed to MCI; the P-values are based on t-tests or chi squares comparing the subjects in the two groups.

Statistical analysis

The overall goal of the time-dependent ROC analyses, as noted above, was to evaluate the prognostic accuracy of the measures described above in predicting which individual subjects developed clinical symptoms consistent with a diagnosis of MCI due to Alzheimer’s disease. These analyses were conducted with three durations of follow-up: 5, 7 and 10 years post-baseline. Four sets of ROC analyses were performed, using these time frames. First, we examined the predictability of the measures for each individual domain. Second, we examined the predictability of all the variables combined (i.e. the Full Model). Third, we sought to determine if a smaller set of measures would have comparable results to the Full Model, reducing the number of variables that would need to be assessed (i.e. referred to here as the Efficient Model); to accomplish this goal, variables were selected on the basis that each one had to be statistically significant when combined together, based on hazard ratios. Lastly, we examined the incremental predictability of variables in models designed to emulate selection criteria that might be used in clinical trials aimed at individuals with preclinical Alzheimer’s disease.

Demographic variables (age, education) were included in all models. For each model mentioned above, we first combined the relevant variables by entering them into a Cox proportional hazards model. The next step was to assess model fit. Cox models were run for every potential combination of variables (among the set of variables chosen for that particular model) in order to determine whether a model with all variables—or a model with a reduced set of variables—produced a better model fit. The Akaike Information Criterion (AIC) was used to compare models to one another to determine the model fit for a given set of biomarker variables and covariates. The AIC was selected for this purpose because it provides an index of the relative balance of model fit (based on the partial likelihood function for the Cox proportional hazards model) and model parsimony (based on the number of parameters in the model). A smaller AIC value indicates a better balance between fit and parsimony (Akaike, 1974).

If the AIC criterion for a set of measures was acceptable (i.e. the difference between the alternate models was <2), then the partial likelihood method for the Cox proportional hazards model was used to create a weighted sum of the measures (the weights being the log hazard ratio corresponding to each measure). Next, the weighted combination of measures from the proportional hazards model was used so that the area under the ROC curve (AUC) was maximized (McIntosh and Pepe, 2002; Blanche et al., 2013). The ROC represents a combined function of the sensitivity (true positive rate) and the specificity (true negative rate) of prediction and the AUC is widely considered a highly informative reflection of a measure(s) overall accuracy for predicting a disease-related outcome. The optimal sensitivity and specificity cut-off point for each model was established by maximizing the Youden index (sensitivity + specificity − 1) (Youden, 1950). The combined set of markers with the higher AUC was considered to be more predictive of disease progression. In this setting the AUC measures the intrinsic ability of the variables to discriminate between participants who developed clinical symptoms and participants who remained normal.

Lastly, different models were compared to one another using point-wise confidence intervals of the AUCs, with confidence intervals constructed using the bootstrap method (Hilbe, 2011). All analyses were implemented in R, Version 3.1.0 (see Supplementary material, section 5 for further details regarding the statistical methods).

Predicting progression from normal cognition to MCI for individual domains

The first set of ROC analyses assessed the predictability of the measures from each individual domain separately (i.e. CSF, MRI, Cognition, APOE status). For each domain, the optimal AIC value (i.e. lowest) was obtained when including both variables from the given domain in the model (after co-varying age and education). As shown in Table 3, in the optimal model for each domain, each of the individual variables were significantly associated with progression from normal cognition to the onset of symptoms of MCI, except for CSF amyloid-β, which was not significant (P = 0.057). Table 3 also shows the AUCs, sensitivities and specificities for the predictability of each domain in relation to the outcome. AUCs were ∼0.70 for all domains, indicating moderate predictive power.

Predicting progression from normal cognition to MCI combining variables from multiple domains

The second set of ROC analyses assessed the accuracy of predicting progression from normal cognition to MCI using the Full Model, which combined all of the variables in the analysis, with no prespecified ordering of the variables. This model was associated with high predictive accuracy (AUC > 0.83 at 5, 7, and 10 years post-baseline) (see Table 4 for the hazard ratios, AUCs, sensitivities and specificities).

We then examined the hazard ratio for each variable in the Full Model to determine if it was statistically significant. The only variable that did not meet these criteria was CSF amyloid-β, as noted above. This variable was therefore excluded, and the Efficient Model was rerun with the remaining variables. The predictive accuracy of the Efficient Model was also high (AUC > 0.82 at 5, 7, and 10 years post-baseline) (Table 4). Statistical comparisons between the Full Model and the Efficient Model revealed no significant differences. Specifically, the high AUCs for both the Full and Efficient Models (>0.83 and >0.82, respectively) did not differ significantly, indicating no difference in predictability (P = 0.53, 0.63, and 0.16, at 5, 7 and 10 years, respectively). This finding is illustrated graphically in Fig. 1, which shows that the Full Model and the Efficient Model have overlapping time-dependent ROC curves at 5 years post-baseline. Likewise, the difference in AIC values between the Full and Efficient Models (393.99 and 394.25, respectively) was <2 at 5 years post-baseline, indicating that the two models were indistinguishable from one another at this time frame (Hilbe, 2011).

Of note, an examination of cases that were misclassified by the Efficient Model at 5 years post-baseline revealed that about 35% of false positive classifications pertained to individuals who progressed at a later time point (mean time from baseline to symptom onset = 7.9 years) (Supplementary material, section 6).

Since the analyses of both the Full Model and the Efficient model demonstrated the importance of demographic variables for accurate prediction (Table 4), we conducted separate ROC analyses with only these variables. The AUCs for demographics alone were approximately 0.68 for all time frames post-baseline (Table 4).

Increment in predictability for variables added in order of potential application in a clinical trial

The last set of ROC analyses examined models designed to emulate an approach to screening that might be used in a clinical trial. In the first model, APOE status was included first, immediately after the demographic variables – this model was designed to emulate the situation in which APOE4 carrier status might be used as an inclusion criterion in a clinical trial aimed at individuals with preclinical Alzheimer’s disease (such as in the API Trial) (Reiman et al., 2011). We then examined the increment in predictability as measures were added consecutively within the model, based on feasibility in a clinical trial, to determine how much each domain adds to the accuracy of prediction at each time point (i.e. APOE status, cognitive measures, MRI measures and CSF measures, adjusted by demographics). Similar to the analyses described above, we continued to require that all variables added to the model had to have a significant hazard ratio and that the model had the smallest AIC compared to alternative models.

Table 5 shows the hazard ratio for each variable when added to the model (as well as the AUCs, sensitivities and specificities) for predicting onset of clinical symptoms at 5, 7 and 10 years post-baseline. Table 5 also shows the P-values comparing the AUCs for models in which we incrementally added variables from the cognitive, MRI and CSF domains, respectively. With only APOE status and demographics in the model, the AUC was approximately 0.70 at 5, 7 and 10 years post-baseline. The addition of the cognitive variables to this model significantly increased the AUC to approximately 0.78 for all follow-up time points (all P < 0.03). The addition of the MRI measures marginally improved predictability above and beyond the genetic and cognitive domains at 5 years (P = 0.052), but the addition of the MRI measures did not significantly improve predictability at 7 and 10 years post-baseline. However, predictability was significantly increased by the addition of CSF p-tau at the last step, increasing AUCs to approximately 0.84 (P < 0.03 for each time point). Figure 2 shows the incremental change in the time-dependent ROC curves predicting the onset of clinical symptoms for variables in this model at 5 years post-baseline. Of note, in a separate set of analyses, using the same model, we added the ratio of CSF p-tau/amyloid-β at the last step (instead of CSF p-tau alone) and found that the addition of this measure did not significantly increase predictability. Moreover, the same finding was true if we added the ratio of CSF.p-tau/amyloid-β at the last step, but APOE4 status was not included at the first step (Supplementary material, section 6).

In the second and third models within this set we focused primarily on the predictability derived from the introduction of the first variable(s) in the model (after the demographics). Since amyloid imaging using PET is currently being used to screen subjects for inclusion in a clinical trial of cognitively normal individuals (e.g. the A4 Study) (Sperling et al., 2014), the second model examined the impact of putting CSF amyloid-β in the model first. With only CSF amyloid-β and demographics in the model, the AUC ranged from 0.70 to 0.72 at 5, 7 and 10 years post-baseline. The sensitivity was 0.64 and the specificity was 0.67 at 5 years. With the addition of other domains to the model, the results were comparable to those in which APOE was added first (Supplementary material, section 7).

In the third model, CSF amyloid-β and CSF p-tau were included first, after the demographic variables—this model was designed to anticipate a future study in which both amyloid imaging and tau imaging might be used to screen subjects for inclusion in a clinical trial aimed at randomizing those who were both amyloid and tau PET positive. With CSF amyloid-β and CSF p-tau in the model (after demographics), the AUC ranged from 0.72 to 0.74 at 5, 7 and 10 years post-baseline (see Table 3, showing model results for the CSF domain). The sensitivity was 0.57 and the specificity was 0.75 at 5 years post-baseline. With the addition of the cognitive and MRI domains to the model, there was an increase in the AUC, with results comparable to those in which either APOE or CSF amyloid-β was added first (data not shown).

As noted above, the models in which two cognitive tests or two MRI variables were included, after the demographic variables, also showed moderate predictability (see Table 3, showing model results for the cognitive and MRI domains). Screening procedures such as these are often considered when more technologically complex and costly methods, such as PET scanning, are not feasible. Of note, though the AUC for both of these models was numerically larger than the one that included APOE (after demographics) (0.76 for cognitive and 0.74 for MRI, versus 0.70 for APOE), the AUC for the models with either the cognitive or the MRI measures included (after demographics) did not differ significantly from the model with APOE (and demographics) at 5 years post-baseline (P = 0.07 and 0.15, respectively), although at 7 and 10 years the difference was significant (P = 0.04 and P = 0.01, respectively).

Limitations

This study has several limitations. It is important to acknowledge that although CSF amyloid-β correlates moderately with PET amyloid levels (Fagan et al., 2006; Vlassenko et al., 2016; Vos et al., 2016), the two are not identical; thus, results may differ if amyloid levels are measured with PET instead of CSF. Likewise, measures of CSF p-tau do not provide information about the regional distribution of tau throughout the brain and tau imaging may therefore provide additional valuable information that can be used in subject selection. Additionally, the use of different MRI measures (i.e. 3 T MRI or other volumetric measures) would likely give different results. The BIOCARD participants were well educated, primarily Caucasian, and the majority had a family history of dementia, so the results may not generalize to the population at large. Participants were also primarily middle-aged when first enrolled, thus the findings may not generalize to older cohorts. Additionally, the relatively small sample size did not allow us to test the reproducibility of these results, and prior work has shown that the specific measures selected may affect diagnostic accuracy (Frisoni et al., 2013). Future studies are therefore necessary to determine if similar findings would be obtained using more diverse groups of older individuals, and different biomarker measures (e.g. 3 T MRI scans; PET imaging for amyloid and tau).