NIH Public Access Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. NIH-PA Author Manuscript Published in final edited form as: J Clin Exp Neuropsychol. 2011 June ; 33(5): 505–522. doi:10.1080/13803395.2010.535504. Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change Lucette A. Cysique1,2, Donald Franklin Jr1, Ian Abramson1, Ronald J. Ellis1, Scott Letendre1, Ann Collier3, Christina Marra3, David Clifford4, Benjamin Gelman5, Justin McArthur6, Susan Morgello7, David Simpson7, J. Allen McCutchan1, Igor Grant1, Robert K. Heaton1, and CHARTER group, and the HNRC group 1HIV Neurobehavioral Research Center (HNRC; http://hnrc.hivresearch.ucsd.edu/), Department of Psychiatry, University of California at San Diego, San Diego, California 2Brain Sciences, School of Psychiatry, Faculty of Medicine, University of New South Wales, Sydney, Australia NIH-PA Author Manuscript 3University of Washington, Seattle; Seattle, WA 4Washington 5University University, St. Louis; St. Louis, MO of Texas Medical Branch; Galveston, TX 6Johns Hopkins University; Baltimore, MD 7Mount Sinai School of Medicine; New York, NY Abstract NIH-PA Author Manuscript Reliable detection and quantification of longitudinal cognitive change are of considerable importance in many neurological disorders, particularly to monitor central nervous system effects of disease progression and treatment. In the current study, we developed normative data for repeated neuropsychological (NP) assessments (6 testings) using a modified Standard RegressionBased (SRB) approach in a sample that includes both HIV-uninfected (HIV−, N=172) and neuromedically stable HIV-infected (HIV+, N=124) individuals. Prior analyzes indicated no differences in NP change between the infected and uninfected participants. The norms for change included correction for factors found to significantly affect follow-up performance, using hierarchical regression. The most robust and consistent predictors of follow-up performance were the prior performance on the same test (which contributed in all models) and a measure of prior overall NP competence (predictor in 97% of all models). Demographic variables were predictors in 10%-46% of all models and in small amounts; while test retest interval contributed in only 6% of all models. Based on the regression equations, standardized change scores (z-scores) were computed for each test measure at each interval; these z scores were then averaged to create a total battery change score. An independent sample of HIV− participants who had completed 8 of the 15 tests was used to validate an abridged summary change score. The normative data are available in an electronic format by email request to the first author. Correction for practice effects based on normative data improved the consistency of NP impairment classification in a clinically stable longitudinal cohort after baseline. Correspondence: Robert K. Heaton: Department of Psychiatry, University of California San Diego, 9500 Gilman Drive #0603, La Jolla, California, 92093-0603. Phone 858-534-4044; Fax: 858-534-9917. rheaton@ucsd.edu. Request of electronic normative data: lcysique@ucsd.edu and specify “request of change normative data” in the subject head of the email. The data will also be available from the HNRC website: http://hnrc.hivresearch.ucsd.edu/ Cysique et al. Page 2 Keywords NIH-PA Author Manuscript Normative data; longitudinal studies; regression; regression change score; SRB; practice effect Introduction The development of norms to reliably identify and quantify neuropsychological (NP) impairment has had a significant impact for clinical and research neuropsychology (Strauss, Sherman, & Spreen, 2006). Normative data that corrects for demographic factors associated with NP performance in healthy controls (i.e., age, education, gender and sometimes ethnicity or pre-morbid ability), enables accurate estimation of disease prevalence, clearer definition of disease and treatment effects and improved clinical management in a variety of neurological and psychiatric disorders. NIH-PA Author Manuscript Neuropsychologists also have emphasised the necessity for providing norms in order to reliably estimate cognitive change over time (Heaton, et al., 2001; Salthouse & TuckerDrob, 2008). Identifying real cognitive change in individuals undergoing repeated NP assessment is potentially valuable for both clinical and research purposes, but diagnosing NP change beyond baseline assessments is difficult because of random biological variation, measurement errors, and practice effects (McCaffrey & Westervelt, 1995) “Practice effect” (PE), or “learning effect” is seen on most NP measures and is larger on measures of problem solving or tests with a high novelty component (Dikmen, et al., 1999). PE complicates detection of meaningful change and leads to underdetection of impairment beyond baseline assessment even when alternate test forms are available. Recent research advances in the area of repeated NP assessment have shown that prediction of cognitive change is substantially improved in clinical samples, including samples for which cognitive change is expected (after treatment for example), when referenced against normative standards. Normative standards are ideally derived from appropriate comparison samples for which no change is expected beyond normal test re-test fluctuation (including practice effect) or regression towards the mean (Heaton, et al., 2001; Salthouse & TuckerDrob, 2008). NIH-PA Author Manuscript One method that has been extensively validated in healthy as well as clinical samples, and that is now commonly used in test development, is the Standard Regression-Based (SRB) change score approach (M. R. Basso, Carona, Lowery, & Axelrod, 2002; Brandt & Benedict, 2001; Chelune, Naugle, Lüders, Sedlak, & Awad, 1993; Dikmen, Heaton, Grant, & Temkin, 1999; Martin, et al., 2002; McSweeny, Naugle, Chelune, & Luders, 1993). This method has an advantage over other change score approaches (Collie, Darby, Falleti, Silbert, & Maruff, 2002) in that it can accommodate multivariate modelling (Temkin, Heaton, Grant, & Dikmen, 1999) with inclusion of numerous factors that may influence NP performance over several assessments. Longitudinal norms for cognitive change using versions of the SRB change score approach have involved widely used NP measures in healthy and relatively young volunteers (Attix, et al., 2009; M. R. Basso, et al., 2002; Dikmen, et al., 1999; Levine, Miller, Becker, Selnes, & Cohen, 2004; Martin, et al., 2002; Temkin, et al., 1999), elderly samples (Duff, et al., 2008; Duff, et al., 2005; Frerichs & Tuokko, 2005; Ivnik, et al., 1999) and samples at risk for cognitive impairment (Heaton, et al., 2001; Hermann, et al., 1996; Sawrie, Chelune, Naugle, & Luders, 1996). Most include normative formulas for individual tests. A few studies have provided norms for scores representing cognitive domains (Duff, et al., 2005) or cognitive J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 3 NIH-PA Author Manuscript factor scores derived from specific NP batteries (Ivnik, et al., 1999), but none of these studies provides guidance to detect and quantify overall cognitive change as derived from a total NP test battery involving multiple cognitive domains. In this regard, the study of Woods, et al. (2006) proposed a battery change score approach using the Reliable Change Index method (RCI). As noted by the authors, a strong advantage of a summary change score approach is to improve the reliability and validity of the observed change as it is based on multiple cognitive abilities. In fact, cognitive change across several NP functions, and in variable patterns, is common in many neurological disorders. NIH-PA Author Manuscript In the present study we have modified the SRB change score approach to allow for: 1) selection of predictors of cognitive change based on previous literature, but with the addition of an estimate of overall prior NP competence and, 2) development of a summary change score approach on the total NP test battery. We developed norms in samples of HIVuninfected (HIV−) people and HIV-infected (HIV+) individuals who were carefully and independently classified as stable with regard to both disease-related and treatment-related neuromedical status. We validated these norms in another sample of HIV− individuals who took a subset of the test battery two or three times. Finally, to enhance accuracy in classifying impairment (versus change) after baseline, we provide normative corrections for practice effects and demonstrate that their use greatly reduces a tendency for clinically stable examinees to appear to have improved abilities over time. The first aim of the current study was to develop normative longitudinal data based on multiple cognitive domains and defined by a single summary change score. To reach this aim, we initially used an HIV-negative (HIV−) sample to develop longitudinal normative SRB equations. The SRB equations were then applied to the clinically stable HIV+ group. The application of the norms to the HIV+ group was done to demonstrate that clinically stable HIV+ do not differ in terms of performance change as compared to HIV− individuals despite a slightly higher rate of baseline impairment rate. NIH-PA Author Manuscript Because the HIV+ and HIV− cohorts showed comparable test-retest results, combining their data for normative purposes provides the advantages of a larger sample size and a broader range of baseline performances from individuals who also share general personal/ background characteristics with the individuals to which the norms for change would be applied (e.g., HIV+ patients who are progressing off treatment, failing their current treatment or initiating new regimens for example). We believe this is the first study to develop norms for change in this context. The normative formulas will be made available in an electronic format, for ease of use. As noted, we also provide a method for correcting for PE on the six repeated assessments. Methods Participants Of 296 volunteers at the San Diego HIV Neurobehavioral Research Center (HNRC) who completed the same test battery over two to six visits, 172 were HIV− controls recruited between 1999 and 2006 and 124 were clinically stable HIV+ individuals recruited through the multi-site CNS HIV Antiretroviral Therapy Effects Research (CHARTER) study between 2001 and 2007. An additional, “validation group” was 111 HIV− volunteers who completed a subset of the same test battery at two time points and 67 who completed the abbreviated battery at three time points at the HNRC between 1987 and 1999. Table 1 compares the demographic characteristics and provide summary results on the NP test battery for the normative HIV− and HIV+ groups and the validation sample. J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 4 NIH-PA Author Manuscript Neuromedical stability was defined for the HIV+ group as having: ( a) stable HIV disease indicators between visits (i.e., CD4 counts not changing among following categories >500, 500–200, <200; < 1 log10 change in their plasma HIV RNA levels (viral load); and no new AIDS defining illnesses), (b) no change in their antiretroviral regimen, and (c) no incident psychiatric illness (i.e., major depressive episode or substance use disorder) or neurological events (i.e., head injury or meningitis) between visits. Participants with a history of non-HIV related neuromedical factors that might potentially cause neurocognitive impairment were excluded. These exclusion criteria were (a) head injury with unconsciousness greater than 30 minutes, (b) any known, non HIV related neurological disorders (e.g., epilepsy, stroke), psychotic disorders (schizophrenia) and (c) significant levels of current self-reported substance use, defined as more than three alcoholic drinks per day over the past 30 days, or use of any illegal drugs in the past 30 days. Diagnosis of bipolar disorder was not disqualifying so long as patients were stable on their medications. Procedure NIH-PA Author Manuscript Neuropsychological Assessments—In the reference sample the NP test battery was composed of 15 individual NP tests which assessed 7 ability areas (see Table 2). When available, alternate versions of the NP tests (i.e., Hopkins Verbal Learning Test-Revised, Brief Visual Memory Test-Revised) were used according to the schedule detailed in the manual to minimize PE. Data Analysis Development of NP norms for change—To identify individuals who presented with overall NP change versus stability (“norms for change”), we used a statistical methodology based on the multivariate Standard Regression-Based (SRB) approach ((McSweeny, et al., 1993; Temkin, et al., 1999; see also Collie, et al., 2002, for review of change score methodologies). The advantage of multiple regression SRB is that it accounts for practice effect, regression towards the mean and other factors that potentially may influence testretest variability in neurologically stable people (e.g., test-retest interval, demographics and “overall NP competence” at baseline) (Temkin, et al., 1999). NIH-PA Author Manuscript Prior to analysis, raw scores for each individual NP measure were transformed into scaled scores (with a mean of 10 and a standard deviation of 3 based on large samples (Heaton et al., 2004; Norman et al, submitted). The scaled score transformations provide a robust method to: a) normalize distributions of multiple NP tests using a common metric, b) attenuate the effects of outliers (Tabachnik & Fidell, 2007). The scaled score transformations will be part of the electronic file that can be requested by email to the first author. The first step in developing norms for change consisted of statistically evaluating the comparability of the HIV+ and HIV− groups to determine whether they could be combined to create a larger reference group for which no NP change is expected beyond practice effect. The 172 HIV-controls were assessed at approximately one-year intervals (14.26 ± 4.89 months) and 124 neuromedically stable HIV+ individuals had been assessed at approximately six month intervals (7.28 ± 2.56 months). Baseline rate of impairment was 22% in the HIV− group and 28% in the HIV+ group (see Table 1) as defined by the Global Deficit Score (GDS) method (Carey et al., 2004). Regression formulas were developed for the HIV− group only, using the method detailed in the following sections for the combined normative data. The change rate in the HIV+ J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 5 NIH-PA Author Manuscript individuals fell well between the 90% prediction intervals of stability based upon regression based norms developed with the 172 HIV-individuals as a reference (see Table 3). In no instance did the resulting rate of change (or no change) of the stable HIV+ group differ significantly from those of the HIV− control sample (all p-values >.20). Consistent with findings from previously published studies, test-retest intervals were not significantly associated with practice effect across testing visits for the current reference samples (Cysique, et al., 2009). Baseline impairment rate in the combined HIV+ and HIV− reference sample reached 24.6%. Table 3 presents the percent rate of change data on the mean scaled score across the study time points for the HIV+ and HIV− samples. NIH-PA Author Manuscript Additionally, we tested whether HIV status was associated with differences in performance change using a multivariate approach, which also provides an adjustment for attrition. For this we developed a mixed effect regression model. HIV status and time were included as fixed linear effects, HIV status and time as a fixed non-linear interaction effect, and subject as a random effect. By incorporating subject-specific effects, i.e. the random intercepts and slopes, the mixed-effects model has the advantage of removing biases in estimation due to attrition at later time points and by adjusting the groups, time and interaction effect intercepts and slopes of the model. We included the model described above (model 1) and a model with demographic and test-retest effect (model 2) in the appendix. In all instances, the HIV status and time interactions were not significant, indicating that there was no difference between the HIV− individuals and HIV+ individuals’ performance changes over time. The entire reference group was therefore composed of 296 volunteers at baseline and at time 2 (test-retest interval: 11.3 months ± 5.3); 241 at time 3 (test-retest interval from baseline: 21.8 months ± 9.0), 171 at time 4 (test-retest interval from baseline: 32.5 months ± 11.0), 132 at time 5 (test-retest interval from baseline: 42.3 months ± 13.2) and 64 at time 6 (testretest interval from baseline: 49.8 months ± 16.5). To investigate differences in performance among subgroups across time, we conducted a series of repeated measure ANOVAs testing for (a) group effect (this effect was represented by including sub-samples according to how many sessions they completed), (b) time effect, and (c) group -by- time interaction effect. The interaction could indicate, for example, that the sample that had completed 6 sessions was different from those who had completed only 2 sessions. In all analyses the interaction term was non-significant (p>.90), indicating that NP performance was homogenous regardless of number of visits completed. Table 4 presents the baseline scaled scores on the 15 NP measures for the reference sample and on the eight NP measures for the validation sample. NIH-PA Author Manuscript To determine which factors influenced longitudinal NP performance in the reference sample, we conducted a series of hierarchical regression analyses using the follow-up scaled score as the outcome variable and a predetermined list of candidate predictors. The selection of these factors and their order as hierarchical predictors of follow-up NP performance was based on previous research literature in the study of longitudinal NP performance (M. Basso, Bornstein, & Lang, 1999; Duff, et al., 2007; Heaton, et al., 2001; Levine, et al., 2004). This step is different from the original method (Chelune, et al., 1993) for which stepwise regression was used. The hierarchical method used the knowledge of previous findings to inform the order of entry into the regression, while stepwise uses only a sample specific statistical cut-off. The predictor variables that were considered in the multivariate analyses included (in order): 1) baseline or previous test performance of the individual NP measure in question (scaled scores), 2) Baseline or previous NP competence (defined as the previous mean scaled score on the battery, with exclusion of the NP measure currently tested in the regression model); J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 6 NIH-PA Author Manuscript 3) age (in years), 4) education (in years); 5) gender (male vs. female coded as 0 and 1), 6) ethnicity (Caucasians vs. Others; coded as 1 and 0), and 7) relevant test-retest interval (in months). Any predictive factor that accounted for significant R2 change above and beyond the previously entered predictors (p<.10) was retained in order to compute reference regression formulas. Table 5 lists which factors were found to be significant predictors of follow-up performance for all the 15 NP measures at all study time points. Multi-colinearity was not detected in any of the models when tolerance statistics and the magnitude of change in the overall R2 were examined for each model after a particular factor was added or excluded. Once the initial predictive models were built, we tested some interaction terms (i.e., overall competence * age, overall competence*previous test score; overall competence * education level, overall competence * sex and age*edu) chosen to detect possible interactions of demographic factors with overall NP competence. From all the models containing the interaction terms (as described above), at most one interaction of predictors emerged as marginally informative for the following NP tests: Letter Fluency and WCST perseverative errors. Accordingly, we decided not to encumber the models by retaining these few interactions. NIH-PA Author Manuscript Next, the significant predictors were retained in a series of standard multivariate regression analyses to derive regression formulas for each individual NP test. These formulas served to compute the 15 individual predicted follow-up scaled scores as shown below [formula 1]: [1] In this formula Yp is the predicted scaled score at follow-up, β1 is the regression coefficient (slope) for predictor X1; β2 for predictor X2; βn for predictor Xn; X1 is the observed baseline or previous test score, X2 and Xn are the overall NP competence, and demographic or re-test interval factors entered into the model and a is the intercept. To inspect for any heteroscedasticity of demographic effects in the resulting 15 models at the six time points (Tabachnick & Fidell, 2007), we conducted a series of Pearson correlation analyses between the residuals of each model and age, education, and gender separately. None of the analyses reached statistical significance (p<.05), showing that demographic effects were quite similar across the NP performance range and over time. NIH-PA Author Manuscript Finally, because most neuropsychological assessments involve the evaluation of several cognitive abilities often yielding a number of individual NP measures, we have developed a method that takes into account all the information within the test battery by computing a battery NP change score or a summary NP change score. Summary regression-based change scores (sRCS) were computed as follows: The standard deviation of the residuals (i.e., error term of the regression model in SD units) for each of the 15 final regression models was computed for time 1 predicting time 2; time 2 predicting time 3; time 3 predicting time 4, time 4 predicting time 5; time 1 predicting time 3 and also predicting time 4, and time 5. Formulas for time 4 predicting time 5 and time 1 predicting time 5 were applied at time 6. This was applied as such because of the relatively small size of the sample at time 6. Altogether, 15 individual z-scores at 6 different time ranges were computed, by dividing the difference between predicted and obtained follow-up scaled scores by the error term of the regression model [see 2]. The resulting Z-score reflects how well or poorly the participant did at follow-up, relative to expectations for a neurocognitively stable person with his/her baseline (or previous) score and other variablespecific baseline predictors: J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 7 [2] NIH-PA Author Manuscript In this formula, Xo is the scaled score at follow-up; Xp is the predicted follow-up scaled score derived from the reference sample regression equations [1], and SDresidual is the standard deviation of the residuals from the reference sample regression model. Note that this regression based change score can be negative (if the obtained follow-up score is worse than the predicted) or positive (if the obtained score is greater than the predicted score). The final step in the development of the norms for change was to average the 15 individual regression based change scores (z-scores), at relevant times, to compute the sRCS and determine a 90% confidence interval to define “no change” on a summary of the entire test battery. Thus, participants in the top 5% of the sRCS distribution of the reference sample were defined as “improved” and the bottom 5% were defined the “decliners”. Cut-offs for significant change are also provided in the appendix for 80% and 70% confidence intervals. NIH-PA Author Manuscript Computation of the Practice effect (PE)—PE corrections were developed separately from the norms for change using scaled scores, on each individual NP measure across the study time points. The median PE of the normative sample was selected for this purpose. That is, the median PE (in scaled score units) was subtracted from the scaled score at followup, to estimate what the performance would have been without practice. We then applied these corrections to the mean scaled score to illustrate how it helps avoid spurious findings of apparent improvement (and decreased sensitivity to impairment or decline) in functioning over repeated NP assessments. Missing data—Of the 18,000 possible scores on every NP test measure across the six testings, 135 (0.75%) were missing. However for each individual, the summary NP performance was composed of at least 12 individual NP measures, in our judgement providing a reliable estimate of global NP performance. NIH-PA Author Manuscript Validation—The norms for change (i.e., prediction formulas from the reference sample regression models and resulting z-scores and sRCS at appropriate times) were then applied to a sample the 111 HIV-persons for validation. These norms were applied to an abridged NP battery consisting of eight of the initial 15 NP tests which were available for the validation sample (Table 2 and Table 3). For this a separate sRCS and confidence interval were computed on the abridged NP battery, using formulas derived from the reference group for the eight relevant NP tests. Test-retest interval was not involved in these analyses because it had no predictive power in the hierarchical models in the reference sample on any of the eight NP measures concerned. Analyses were conducted using PAWS 18.0 version, JMP 7.0 version (SAS Inc) and the effect size calculator from (Lipsey & Wilson, 2001). Results Reliability and practice effect (PE) findings Table 6 presents median overall PE computed from scaled scores (i.e., using the same metric across all NP measures). We retained the “median PE” rather than the alternative “average PE” because it provided a better estimate of overall stability. Indeed, the average PE across all 15 measures tended to overestimate PE in the mean scaled score (the most reliable measure; see Table 7) across time, while the median PE correction better matched the goal of “no change”. J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 8 NIH-PA Author Manuscript In Figure 1, we illustrate the application of these PE corrections on the mean scaled score for the sample re-tested five times after baseline (see also Table 7 which provides this for all time points). Without correction, an apparent improvement in the mean scaled score is observable at each session reaching almost a full scaled score point at time 6. PEs were found to be larger, in most instances, at the first test-retest interval (sum corrections = 6 scaled score points), but considerable additional PE also occurred from time 2 to time 5 (sum corrections = 6.5 additional scaled score points, see Table 5 and Figure 1). The median PE figures did not provide any additional correction between time 5 and time 6 (see sum in Table 5 suggesting that PE is subtle or variable thereafter). Table 8 presents reliability statistics in the reference sample. Test re-test reliability was overall adequate. Median correlation coefficients (r’s) across the 15-test battery were .67 for time 1 – 2; .68 for time 2 – 3; .70 for time 3 – 4; .69 for time 4 – 5, and .68 for time 5 – 6. The highest test retest reliability was observed for the mean scaled scores (.88 – .92), and for certain speed of information processing tests and working memory tests (Digit Symbol: .89; PASAT-50: .82). The reliability was lowest for the delayed recall in memory tests (medians of .52 and .63). The use of scaled scores rather than raw scores improved the reliability statistic on several tests, while in other tests the difference with the raw scores was small in either direction (data not shown). NIH-PA Author Manuscript Validation of the sRCS In the validation sample, classification of cases as improving or declining at time 2 or 3 did not differ from the 5% reference sample’s prediction (Table 9). When considering the continuous sRCS, the validation sample declined more than the reference sample only at time 2. While this represented a significant statistical difference, we determined that this average decline (−0.16) was far from the confidence interval boundary for significant decline, that is > 0.825. Thus, use of confidence intervals to determine a significant and clinically meaningful level of change at the individual level remains quite accurate; that is, the “percent declined” in this validation sample at time 2 is only 6% (very close to the 5% in the reference sample) and the maximum deviation from expected across these validation points was two percent (Table 8). Further analyses using each of the eight individual NP measures yielded somewhat different proportions of stability vs. change (improvement or decline) especially on the Trail Making Test part B. Thus, targeting the summary measure of cognition for interpretation of meaningful change (here the sRCS) approach, while most conservative, is also more reliable and should be recommended over using the SRB approach on individual tests. NIH-PA Author Manuscript Finally, we compared the continuous and the discrete predictions of change in the reference sample using the 15 NP test-based sRCS versus the 8 NP test-based sRCS. We found for time 1 – 2 a 90.5% agreement in the classification of change and a significant correlation for the continuous sRCS (r= .77; p<.0001); for time 2 – time 3 there was an 88.0% agreement in the classification of change and a significant correlation for the continuous sRCS (r= .79; p<.0001); and for time 1 – 3 there was a 91.0% agreement in the classification of change and a significant correlation for the continuous sRCS (r= .81; p<.0001). In all instances, disagreement in change classification involved “change” versus “no change” (never change in the opposite direction). Discussion In the current study we developed a series of summary regression-based change scores (sRCS) over six NP testings in a sample of 296 HIV-uninfected and neuromedically stable HIV-infected individuals. These longitudinal normative standards - in an abridged format J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 9 NIH-PA Author Manuscript were then successfully applied to validation samples of 111 HIV-uninfected individuals who were assessed twice or 3 times (N=67) based on 8 NP tests. Estimates of the proportion of individuals with significant NP performance change did not differ between the normative sample and the validation sample, in both groups approximately 5% improved or declined and 90% were stable. We also found that the 15 NP test-based norms and the 8 test-based norms were highly correlated demonstrating a high level of agreement in the classification of “change” or “no change”. NIH-PA Author Manuscript Prior to the development of our norms we have demonstrated, using two different statistical methods (the SRB method and mixed effect regression analyses) that clinically stable HIV+ individuals perform similarly to HIV− individuals in terms of test-retest change despite a slightly higher impairment rate at baseline in the HIV+ sample. Our study confirms and extends our previous findings (Cysique et al., 2009) where we had found that clinically stable HIV+ individuals performed similarly to HIV− individuals over a one-year period. Here we show that this longitudinal comparability extended to a three-year period, and was even seen when we adjust for attrition as allowed mixed effect regression analyses. Another large study has found that clinically stable HIV+ individuals had stable psychomotor speed performance over a five-year period as compared to HIV− individuals (Cole et al., 2007). There is therefore cumulative evidence that the stability of performance across several years is associated with stable HIV disease, providing a valuable pool of performance data that can be standardised use to create norms for change. While this applies to HIV infection, where long-term treatment is known to be increasingly effective and increasing patients’ life expectancies almost as long as non-HIV infected individuals, we would recommend as proposed here, that the stability of performance is empirically determined (by comparison to HIV− controls) prior to normative data development. NIH-PA Author Manuscript Moreover it should be noted that the HIV+ reference sample represents a carefully selected subgroup of the CHARTER cohort, who have both minimal confounds and very stable disease and treatment over time as well as no incidence of psychiatric conditions and substance use disorders. Because of this, their baseline impairment rate is much lower than the rest of CHARTER cohort (28% versus 52% overall and 40% in the total subgroup with minimal comorbidity (Heaton et al. press). This explains why we did not find a statistically significant difference (at p<.05) in baseline impairment and overall NP performance on the GDS when compared to the HIV− reference sample (only a trend; p<.08). Concerning the HIV− reference sample, it should be noted that in the current era, HIV− participants in HNRC studies (those who have the current NP battery) share lifestyle factors with the HIV+ reference sample (e.g., they do not have current or incident substance use disorders but could have lifetime disorders). Our combined HIV+ and HIV− sample provides larger age and educational ranges, as well as broader gender and ethnicity representations (see Table 1) and increase range of baseline NP performance. All those characteristics are not only valuable for HIV research, but also beyond this field of research, as the methods should be applicable to a range of neuromedical conditions. Further inspection of our normative data demonstrated that our norms for change were independent of baseline impairment. Indeed, the sRCS were at all times not statistically different between impaired and unimpaired individuals at baseline (dichotomous sRCS p>.28). When using the continuous sRCS, there was no systematic indication that the impaired individuals declined or improved as a group as compared to the unimpaired group. In addition to this evidence of validity of the current approach in clinically stable HIV+ patients and HIV-controls, the same methods were found to be valid for detecting cognitive changes produced by HIV-associated disease and a CNS penetration of antiretrovirals in another study (Cysique, et al., 2009). We have also used the same sRCS methodology in J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 10 cross-cultural settings, finding that cognitive decline measured in this way was sensitive to HIV disease-related change in China (Cysique, et al., 2010). NIH-PA Author Manuscript Another method for detecting meaningful cognitive change is based on PRESS statistics (Berres, Zehnder, Blasi, & Monsch, 2008.; Blasi, et al., 2009). Briefly, this method reportedly provides an improved selection of prediction models compared to the stepwise regression often used prior the generation of SRB change scores. Interestingly, several of their models include interaction terms involving demographics. In our study, we found that the inclusion of interaction terms did not substantially improve our predictive models. Some important differences between the methods in the two studies include that the PRESS-based method was validated on only one test of learning and memory (i.e., the California Verbal Learning Test) and did not account for overall NP competence as we did (of course, our measure of prior NP competence requires longitudinal use of a test battery, rather than a single test). Without the inclusion of overall NP competence, a significant amount of variance may be unaccounted in predicting NP change. This may explain why the addition of interaction terms is important in the PRESS studies, but not in ours. In future studies use of the exact same predictive models (models inclusive of overall NP competence as a predictor when using battery wide summary change scores) might be compared for the SRB method and the PRESS method. NIH-PA Author Manuscript With respect to our inclusion of overall NP competence, one might assume that a pre-morbid ability estimate (e.g. pre-morbid IQ) might perform better than current cognitive functioning. However, in one prior study, it has been shown that a measure of premorbid functioning (estimated IQ) did not independently predict meaningful cognitive change (Frerichs & Tuokko, 2005). NIH-PA Author Manuscript It might be suggested that the construction of our SRB formula privileged the prior performance in a test and the overall NP competence as they were entered as first and second predictors in the hierarchical model which eventually served to select the predictors to be included in the final formulas. In fact, we found that overall NP competence was a significant predictor in 97% of all models, while demographic factors were less frequently significant (see Table 5; age is a predictor in 46% of all models; education is a predictor in 15% of all models, gender is a predictor in 13% of all models; ethnicity is a predictor in 10% of all models; and test-rested interval is a predictor in only 6% of all models). In addition, the variance explained by both the prior performance on the same test and the overall NP competence contributed to 70–90% of the overall model R2; while demographic factors accounted for between 5–15% of explained variance, and test retest-interval to 1.5–3%. Further comparison of our “order of entry” of demographics confirmed that the variance explained by demographics was lessened in our approach, but that our final models were simpler and less prone to apparently spurious associations (e.g., predictions going in inconsistent directions across the test variables; see Table A2 in the appendix). While overall test-retest reliability was adequate for individual tests, the highest reliability was found for the mean scaled scores and some psychomotor speed and working memory tests. Lowest reliability was found for some delayed recall and executive functions measures. These patterns are similar to findings of previous studies (M. Basso, et al., 1999; Temkin, et al., 1999). Higher reliability in psychomotor speed based tests is known to be boosted by their psychometric properties such as large range of possible values and approximation to a normal distribution even on raw scores. In contrast, measures of memory and executive functions often have a restricted range of possible values, especially in follow-up assessments. They are also intrinsically more susceptible to learning effects (M. Basso, et al., 1999; Temkin, et al., 1999). We found the improved distributions provided by the scaled scores enhanced the test-retest reliability of those measures and did not J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 11 NIH-PA Author Manuscript substantially change those of other NP measures. In the case of the memory tests, alternate versions were used for each of the five follow-ups, perhaps contributing to reduced testretest reliability. Practice effects were found to be larger, in most instances, in the first test-retest interval in accordance with previous observations (Collie, et al., 2003). However, additional PEs were found at visits three to five for multiple test measures and in particular on the WCST, demonstrating that learning continues over multiple exposures to tests of executive functioning (M. Basso, et al., 1999; Levine, et al., 2004). Our study had several limitations: 1) The reference group was mainly male, but with a relatively good age range (18–66) and education range (7–20), and was diverse in terms of ethnicity representation (Table 1). Although age and gender were not strong predictors of test-battery test change in our models, our results maybe less generalizable to women and older individuals (beyond age 65). NIH-PA Author Manuscript 2) The sRCS method is relatively complex, requiring several computational steps. To aid potential users, we have built a password protected spreadsheet which will include the needed raw to scaled scores transformations, the computation of the z-scores and the sRCS with the normative confidence intervals. Only the final computed sRCS will appear on this spreadsheet and the entry field of the relevant predictor for each of the 15 NP measures. The formulas will be password protected and accessible in plain text format upon request to the first author. The spreadsheet is available upon request to the first author (lcysique@ucsd.edu) and will be held by the HIV Neurobehavioral Research Center at San Diego, USA < http://hnrc.hivresearch.ucsd.edu/>. 3) The NP testing required is relatively extensive, requiring approximately 2 hours to complete. The optimal use of the current sRCS would require the use of the 15 NP measures in the version presented in Table 3. However, the abridged sRCS based on 8 NP measures might be used as a second choice as we have shown that it has substantial overlap with the 15 NP measure-based sRCS predictions. NIH-PA Author Manuscript In conclusion, we provide norms for a moderately comprehensive test battery, using an improved method to quantify global cognitive change that can be applied in clinical samples to determine disease-related incidence of cognitive decline or incidence of cognitive improvement after treatment initiation. In the future, the summary sRCS approach should be compared to other recently developed methods for interpretation of cognitive change and to determine how well they perform overall and within specific cognitive domains. Supplementary Material Refer to Web version on PubMed Central for supplementary material. Acknowledgments This study was supported by the HIV Neurobehavioral Research Center (HNRC* supported by Center award MH 62512 from NIMH), the CNS HIV Anti-Retroviral Therapy Effects Research (CHARTER**) study (supported by award N01 MH22005 from the National Institutes of Health), by the MH 58076 grant, and Brain Sciences postdoctoral fellowship at the University of New South Wales. Dr. Cysique is supported by the Brain Sciences post-doctoral fellowship from the University of New South Wales, Sydney, Australia. J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 12 References NIH-PA Author Manuscript NIH-PA Author Manuscript Attix DK, Story TJ, Chelune GJ, Ball JD, Stutts ML, Hart RP, et al. The prediction of change: normative neuropsychological trajectories. The Clinical Neuropsychologist. 2009; 23(1):21–38. [PubMed: 18720272] Basso M, Bornstein R, Lang J. Practice effects on commonly used measures of executive function across twelve months. The Clinical Neuropsychologist. 1999; 13(3):283–292. [PubMed: 10726600] Basso MR, Carona FD, Lowery N, Axelrod BN. Practice effects on the WAIS-III across 3- and 6month intervals. The Clinical Neuropsychologist. 2002; 16(1):57–63. [PubMed: 11992227] Berres M, Zehnder A, Blasi S, Monsch AU. Evaluation of diagnostic scores with adjustment for covariates. Stat Med. 2008; 27(10):1777–1790. [PubMed: 17968872] Blasi S, Zehnder AE, Berres M, Taylor KI, Spiegel R, Monsch AU. Norms for change in episodic memory as a prerequisite for the diagnosis of mild cognitive impairment (MCI). Neuropsychology. 2009; 23(2):189–200. [PubMed: 19254092] Brandt, J.; Benedict, RHB. Hopkins Verbal Learning Test-Revised. Lutz, FL: Psychological Assessment Resources, Inc; 2001. Carey CL, Woods SP, Gonzalez R, Conover E, Marcotte TD, Grant I, et al. Predictive validity of global deficit scores in detecting neuropsychological impairment in HIV infection. Journal of Clinical and Experimental Neuropsychology. 2004; 26(3):307–319. [PubMed: 15512922] Chelune GJ, Naugle RI, Lüders H, Sedlak J, Awad IA. Individual change following epilepsy surgery: Practice effects and base-rate information. Neuropsychology. 1993; 1:41–52. Cole MA, Margolick JB, Cox C, Li X, Selnes OA, Martin EM, et al. Longitudinally preserved psychomotor performance in long-term asymptomatic HIV-infected individuals. Neurology. 2007; 69(24):2213–2220. [PubMed: 17914066] NIH-PA Author Manuscript *The San Diego HIV Neurobehavioral Research Center [HNRC] group is affiliated with the University of California, San Diego, the Naval Hospital, San Diego, and the Veterans Affairs San Diego Healthcare System, and includes: Director: Igor Grant, M.D.; CoDirectors: J. Hampton Atkinson, M.D., Ronald J. Ellis, M.D., Ph.D., and J. Allen McCutchan, M.D.; Center Manager: Thomas D. Marcotte, Ph.D.; Naval Hospital San Diego: Braden R. Hale, M.D., M.P.H. (P.I.); Neuromedical Component: Ronald J. Ellis, M.D., Ph.D. (P.I.), J. Allen McCutchan, M.D., Scott Letendre, M.D., Edmund Capparelli, Pharm..D., Rachel Schrier, Ph.D.; Neurobehavioral Component: Robert K. Heaton, Ph.D. (P.I.), Mariana Cherner, Ph.D., Steven Paul Woods, Psy.D.; Neuroimaging Component: Terry Jernigan, Ph.D. (P.I.), Christine Fennema-Notestine, Ph.D., Sarah L., Archibald, M.A., John Hesselink, M.D., Jacopo Annese, Ph.D., Michael J. Taylor, Ph.D., Neurobiology Component: Eliezer Masliah, M.D. (P.I.), Ian Everall, FRCPsych., FRCPath., Ph.D.,Cristian Achim, M.D., Ph.D..; Neurovirology Component: Douglas Richman, M.D., (P.I.), David M. Smith, M.D.; International Component: J. Allen McCutchan, M.D., (P.I.); Developmental Component: Ian Everall, FRCPsych., FRCPath., Ph.D (P.I.), Stuart Lipton, M.D., Ph.D.; Clinical Trials Component: J. Allen McCutchan, M.D., J. Hampton Atkinson, M.D., Ronald J. Ellis, M.D., Ph.D., Scott Letendre, M.D.; Participant Accrual and Retention Unit: J. Hampton Atkinson, M.D. (P.I.), Rodney von Jaeger, M.P.H.; Data Management Unit: Anthony C. Gamst, Ph.D. (P.I.), Clint Cushman, B.A., (Data Systems Manager), Daniel R. Masys, M.D. (Senior Consultant); Statistics Unit: Ian Abramson, Ph.D. (P.I.), Christopher Ake, Ph.D., Florin Vaida, Ph.D. The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of the Navy, Department of Defense, nor the United States Government. **The CNS HIV Anti-Retroviral Therapy Effects Research (CHARTER) group is affiliated with the Johns Hopkins University, Mount Sinai School of Medicine, University of California, San Diego, University of Texas, Galveston, University of Washington, Seattle, Washington University, St. Louis and is headquartered at the University of California, San Diego and includes: Director: Igor Grant, M.D.; Co-Directors: J. Allen McCutchan, M.D., Ronald J. Ellis, M.D., Ph.D., Thomas D. Marcotte, Ph.D.; Center Manager: Donald Franklin, Jr.; Neuromedical Component: Ronald J. Ellis, M.D., Ph.D. (P.I.), J. Allen McCutchan, M.D., Terry Alexander, R.N.; Laboratory, Pharmacology and Immunology Component: Scott Letendre, M.D. (P.I.), Edmund Capparelli, Pharm.D.; Neurobehavioral Component: Robert K. Heaton, Ph.D. (P.I.), J. Hampton Atkinson, M.D., Steven Paul Woods, Psy.D., Matthew Dawson; Virology Component: Joseph K. Wong, M.D. (P.I.); Imaging Component: Christine Fennema-Notestine, Ph.D. (P.I.), Terry L., Jernigan, Ph.D., Michael J. Taylor, Ph.D., Rebecca Theilmann, Ph.D.; Data Management Unit: Anthony C. Gamst, Ph.D. (P.I.), Clint Cushman,; Statistics Unit: Ian Abramson, Ph.D. (P.I.), Florin Vaida, Ph.D.; Protocol Coordinating Component: Thomas D. Marcotte, Ph.D. (P.I.), Rodney von Jaeger, M.P.H.; Johns Hopkins University Site: Justin McArthur (P.I.), Mary Smith; Mount Sinai School of Medicine Site: Susan Morgello, M.D. (Co-P.I.) and David Simpson, M.D. (Co-P.I.), Letty Mintz, N.P.; University of California, San Diego Site: J. Allen McCutchan, M.D. (P.I.), Will Toperoff, N.P..; University of Washington, Seattle Site: Ann Collier, M.D. (Co-P.I.) and Christina Marra, M.D. (Co-P.I.), Trudy Jones, M.N., A.R.N.P.; University of Texas, Galveston Site: Benjamin Gelman, M.D., Ph.D. (P.I.), Eleanor Head, R.N., B.S.N.; and Washington University, St. Louis Site: David Clifford, M.D. (P.I.), Muhammad Al-Lozi, M.D., Mengesha Teshome, M.D. The views expressed in this article are those of the authors and do not reflect the official policy or position of the United States Government. J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 13 NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Collie A, Darby DG, Falleti MG, Silbert BS, Maruff P. Determining the extent of cognitive change after coronary surgery: a review of statistical procedure. Annals of Thoracic Surgery. 2002; 73:2005–2011. [PubMed: 12078822] Collie A, Maruff P, Darby D, McStephen M. The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. Journal of the International Neuropsychological Society. 2003; 9(3):419–428. [PubMed: 12666766] Cysique LA, Letendre SL, Ake C, Jin H, Franklin DR, Gupta S, et al. Incidence and nature of cognitive decline over 1 year among HIV-infected former plasma donors in China. AIDS. 2010; 24(7):983–990. [PubMed: 20299964] Cysique LA, Vaida F, Letendre S, Gibson S, Cherner M, Woods SP, et al. Dynamics of cognitive change in impaired HIV-positive patients initiating antiretroviral therapy. Neurology. 2009; 73(5): 342–348. [PubMed: 19474412] Dikmen SS, Heaton RK, Grant I, Temkin NR. Test-retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery. Journal of the International Neuropsychological Society. 1999; 5(4):346–356. [PubMed: 10349297] Duff K, Beglinger LJ, Schultz SK, Moser DJ, McCaffrey RJ, Haase RF, et al. Practice effects in the prediction of long-term cognitive outcome in three patient samples: a novel prognostic index. Archives of Clinical Neuropsychology. 2007; 22(1):15–24. [PubMed: 17142007] Duff K, Beglinger LJ, Van Der Heiden S, Moser DJ, Arndt S, Schultz SK, et al. Short-term practice effects in amnestic mild cognitive impairment: implications for diagnosis and treatment. International Psychogeriatrics. 2008; 20(5):986–999. [PubMed: 18405398] Duff K, Schoenberg MR, Patton D, Paulsen JS, Bayless JD, Mold J, et al. Regression-based formulas for predicting change in RBANS subtests with older adults. Archives of Clinical Neuropsychology. 2005; 20(3):281–290. [PubMed: 15797165] Frerichs RJ, Tuokko HA. A comparison of methods for measuring cognitive change in older adults. Archives of Clinical Neuropsychology. 2005; 20(3):321–333. [PubMed: 15797168] Heaton R, Temkin N, Dikmen S, Avitable N, Taylor M, Marcotte T, et al. Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of Clinical Neuropsychology. 2001; 16(1):75–91. [PubMed: 14590193] Heaton, RK.; Miller, SW.; Taylor, MJ.; Grant, I. Revised comprehensive norms for an expanded Halstead-Reitan Battery: Demographically adjusted neuropsychological norms for African American and Caucasian adults Scoring Program. Odessa: FL: Psychological Assessment Resources; 2004. Heaton RK, Clifford D, Franklin DR, Woods SP, Ake C, Vaida F, et al. HIV-associated neurocognitive disorders persist in the era of potent antiretroviral therapy: CHARTER Study. Neurology. (In press). Hermann BP, Seidenberg M, Schoenfeld J, Peterson J, Leveroni C, et al. Empirical techniques for determining the reliability, magnitude, and pattern of neuropsychological change after epilepsy surgery. Epilepsia. 1996; 37(10):942–950. [PubMed: 8822692] Ivnik RJ, Smith GE, Lucas JA, Petersen RC, Boeve BF, Kokmen E, et al. Testing normal older people three or four times at 1- to 2-year intervals: Defining normal variance. Neuropsychology. 1999; 13(1):121–127. [PubMed: 10067783] Levine A, Miller E, Becker J, Selnes O, Cohen BA. Normative data for determining significance of test-retest differences on eight common neuropsychological instruments. The Clinical Neuropsychologist. 2004; 18:373–384. [PubMed: 15739809] Lipsey, M.; Wilson, D. Practical meta-analysis. London: Sage Publications; 2001. Martin R, Sawrie S, Gilliam F, Mackey M, Faught E, Knowlton R, et al. Determining reliable cognitive change after epilepsy surgery: development of reliable change indices and standardized regression-based change norms for the WMS-III and WAIS-III. Epilepsia. 2002; 43(12):1551– 1558. [PubMed: 12460258] McCaffrey R, Westervelt H. Issues associated with repeated neuropsychological assessment. Neuropsychology Review. 1995; 5(3):203–221. [PubMed: 8653109] J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 14 NIH-PA Author Manuscript NIH-PA Author Manuscript McSweeny AJ, Naugle RI, Chelune GJ, Luders H. “T scores for change”: An illustration of a regression approach to depicting change in clinical neuropsychology. The Clinical Neuropsychologist. 1993; 7:300–312. Norman MA, Moore DJ, Taylor M, Franklin D, Cysique LA, Ake C, et al. Demographically Corrected Norms for African Americans and Caucasians on the Hopkins Verbal Learning Test-Revised, Brief Visuospatial Memory Test-Revised, Stroop Color and Word Test, and Wisconsin Card Sorting Test 64-Card Version. Journal of Clinical and Experimental Neuropsychology. (Submitted). Salthouse TA, Tucker-Drob EM. Implications of short-term retest effects for the interpretation of longitudinal change. Neuropsychology. 2008; 22(6):800–811. [PubMed: 18999354] Sawrie SM, Chelune GJ, Naugle RI, Luders HO. Empirical methods for assessing meaningful neuropsychological change following epilepsy surgery. Journal of the International Neuropsychological Society. 1996; 2(6):556–564. [PubMed: 9375160] Strauss, E.; Sherman, EMS.; Spreen, O. A Compendium Of Neuropsychological Tests: Administration, Norms, And Commentary. 3rd ed.. Oxford: Oxford University Press; 2006. Tabachnik, BG.; Fidell, LS. Using Multivariate Statistics. 5th ed.. Boston: Person International Edition; 2007. Temkin N, Heaton R, Grant I, Dikmen S. Detecting significant change in neuropsychological test performance: a comparison of four models. Journal of the International Neuropsychological Society. 1999; 5(4):357–369. [PubMed: 10349298] Woods SP, Childers M, Ellis RJ, Guaman S, Grant I, Heaton RK. A battery approach for measuring neuropsychological change. Archives of Clinical Neuropsychology. 2006; 21(1):83–89. [PubMed: 16169705] NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Cysique et al. Page 15 NIH-PA Author Manuscript Figure 1. NIH-PA Author Manuscript NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 1 J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. HIV− Reference Sample (N=172) HIV+ Reference Sample (N=124) Total Reference Sample (HIV− & HIV+) (N=296) HIV− Validation Sample (N=111) P1 P2 Age 37.86 (11.30) 43.93 (8.89) 40.40 (10.8) 33.95 (8.12) <.0001 <.0001 Education 12.92 (2.35) 12.85 (2.27) 12.9 (2.32) 14.30 (2.57) .80 <.0001 64.5% 78.2% 70% 77% .01 .19 Gender (% male) Ethnicity (% Caucasian) 68% 44% 58% 66% <.0001 .19 % HIV+ clinically stable - 100% 41.8% 0% - - % Baseline NP impairment 22% 28% 25% 15% .22 .05 Overall impairment (GDS) 0.32 (0.38) 0.41 (0.48) 0.36 (0.43) 0.29 (0.32) .08 .08 Cysique et al. Demographic and clinical characteristics of the study samples GDS: Global Deficit Score P-value1: comparisons between the HIV+ and HIV− samples P-value2: comparisons between the reference and the validation samples Page 16 Cysique et al. Page 17 Table 2 Neuropsychological Test Battery NIH-PA Author Manuscript Verbal Fluency Animal fluency * (Benton, Hamsher & Sivan, 1994; Heaton et al., 2004) Letter fluency * (Benton, Hamsher & Sivan, 1994; Gladsjo et al., 1999) Attention/Working Memory PASAT-50 * (Gronwall, 1997; Diehr, Heaton & Miller, 1998) WAIS-III L-N Sequencing (Wechsler Adult Intelligence Scale-Third Edition, Psychological Corporation, 1997; Heaton, Taylor & Manly, 2003) Speed of Information Processing WAIS-III Digit Symbol * (Wechsler Adult Intelligence Scale-Third Edition, Psychological Corporation, 1997, Heaton, Taylor & Manly, 2003) WAIS-III Symbol Search (Wechsler Adult Intelligence Scale-Third Edition, Psychological Corporation, 1997, Heaton, Taylor & Manly, 2003) Trail Making Test A (Army Individual Test Battery, 1944) * NIH-PA Author Manuscript Executive Functioning WCST-64 (Kongs, Thompson, Iverson, & Heaton, 2000) Trail Making Test B (Army Individual Test Battery, 1944) * Learning/Memory Verbal (Hopkins Verbal Learning Test-Revised) total learning & delayed recall (Brandt & Benedict, 2001) Visual (Brief Visuospatial Memory Test-Revised) total learning & delayed recall (Benedict, 1997) Motor Grooved Pegboard dominant * & non-dominant hand * (Klove, 1963; Heaton et al., 2004) * NP measures included in the abridged summary score for cross-validation. NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 3 % decline Times % stable % improve Continuous sRCS J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. HIV− Reference Sample HIV+ Reference Sample HIV− Reference Sample HIV+ Reference Sample HIV− Reference Sample HIV+ Reference Sample HIV− Reference Sample HIV+ Reference Sample p p Time 1 pred Time 2 4.65% 3.23% 90.70% 93.55% 4.65% 3.23% .67 − 0.00 (1.00) 0.09 (0.90) .39 Time 2 pred Time 3 5.13% 1.61% 89.74% 91.94% 5.13% 6.45% .29 − 0.00 (1.00) 0.01 (0.95) .94 Time 3 pred Time 4 4.30% 2.56% 91.40% 96.15% 4.30% 1.28% .40 − 0.00 (1.00) − 0.10 (0.95) .50 Time 4 pred Time 5 4.23% 6.56% 91.55% 88.52% 4.23% 4.92% .81 − 0.00 (1.00) −0.02 (1.28) .90 Cysique et al. Rate of significant change in the HIV+ and HIV− reference samples, for the mean scaled score (90% confidence interval sRCS cut-offs), and continuous sRCS, results across study time points Summary regression-based change scores (sRCS) Hierarchical models were developed in the 172 HIV− individuals only with previous performance in 1st order of entry and then demographic characteristics. The predictor’s model at time 1 predicting time 2 included, the time 1 mean scaled score and age; the predictor’s model at time 2 predicting time 3 included, the time 2 mean scaled score; the predictor’s model at time 3 predicting time 4 included, the time 3 mean scaled score; the predictor’s model at time 4 predicting time 5 included, the time 4 mean scaled score and education. The SRB equations were then built and applied to the HIV+ individuals. Page 18 Cysique et al. Page 19 Table 4 NIH-PA Author Manuscript Baseline NP performance (scaled scores, mean (SD)) on 15 NP measures in the HIV− and HIV+ reference samples (separately and combined) and in the validation samples NIH-PA Author Manuscript HIV− Reference Sample (N=172) HIV+ Reference Sample (N=124) Total Reference Sample (N=296) HIV− Validation Sample (N=111) Letter fluency (total correct) 10.61 (2.65) 9.72 (2.73) 10.24 (2.72) 11.23 (2.74) Animal Fluency (total correct) 11.03 (2.34) 9.77 (2.51) 10.5 (2.49) 11.45 (2.56) PASAT 50 (total correct) 9.76 (2.64) 8.27 (3.11) 9.13 (2.94) 10.24 (2.70) WAIS-III L-N Sequencing (total correct) 10.39 (2.61) 9.45 (2.72) 9.98 (2.69) - WAIS-III Digit Symbol (total correct) 9.68 (2.69) 10.08 (2.89) 9.85 (2.78) 10.17 (2.66) WAIS-III Symbol Search (total correct) 10.28 (2.64) 9.17 (3.05) 9.81 (2.87) - Trail Making Test A (time in seconds) 10.91 (2.32) 9.24 (2.25) 10.20 (2.43) 11.31 (2.69) WCST-64 (perseverative errors) 9.00 (3.08) 7.07 (2.79) 8.19 (3.11) - Trail Making Test B (time in seconds) 10.99 (2.53) 9.64 (2.99) 10.42 (2.81) 11.63 (2.55) HVLT-R Total Learning (total correct) 8.87 (2.71) 8.08 (2.92) 8.54 (2.83) - HVLT-R Delayed Recall (total correct) 8.96 (3.02) 7.60 (3.36) 8.39 (3.23) - BVMT Total Learning (total correct) 9.24 (2.93) 8.05 (2.92) 8.74 (2.98) - BVMT Delayed Recall (total correct) 9.07 (3.09) 7.84 (3.07) 8.56 (3.13) - Grooved Pegboard DH (time in seconds) 9.93 (2.67) 8.66 (2.88) 9.40 (2.82) 10.66 (2.28) Grooved Pegboard NDH (time in seconds) 9.74 (2.61) 8.24 (2.94) 9.15 (2.78) 10.31 (2.40) L-N: Letter-Number; WCST-64: Wisconsin Card Sorting Test; HVLT-R: Hopkins Verbal Learning Test-Revised; BVMT: Brief Visuospatial Memory Test-Revised; DH: Dominant hand; NDH, non-dominant hand NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 5 Overall competence Time 1 pred time 2 + + Time 2 pred time 3 + + .61 Time 3 pred time 4 + + .63 Time 4 pred time 5 + + .58 Time 1 pred time 3 + Time 1 pred time 4 + Time 1 pred time 5 + Age Education Sex Ethnicity Test-retest interval R2 final model Previous score Cysique et al. Significant predictors (of change, p<.10) of follow-up performance on the 15 NP measures in the final hierarchical models (scaled scores) Letter Fluency + .59 J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. .60 − .63 .49 Animal Fluency Time 1 pred time 2 + + .48 Time 2 pred time 3 + + .46 Time 3 pred time 4 + + Time 4 pred time 5 + + Time 1 pred time 3 + + Time 1 pred time 4 + + Time 1 pred time 5 + + Time 1 pred time 2 + + Time 2 pred time 3 + + Time 3 pred time 4 + + Time 4 pred time 5 + + Time 1 pred time 3 + + − Time 1 pred time 4 + + − Time 1 pred time 5 + + − + + .53 + .50 .33 − .43 + .46 PASAT 50 − + .68 − .71 + .73 − .72 − .65 .61 − .60 WAIS III L-N sequencing + .45 Page 20 Time 1 pred time 2 NIH-PA Author Manuscript NIH-PA Author Manuscript Time 2 pred time 3 + + Time 3 pred time 4 + + Time 4 pred time 5 + + Time 1 pred time 3 + + − .45 Time 1 pred time 4 + + − .50 Time 1 pred time 5 + + − .44 + + − + .70 Time 2 pred time 3 + + − Time 3 pred time 4 + + Time 4 pred time 5 + + Time 1 pred time 3 + + Education Sex Ethnicity − .52 + .45 .42 Cysique et al. NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Overall competence Age Test-retest interval R2 final model Previous score WAIS III Digit symbol Time 1 pred time 2 .74 + + .80 .84 + .68 Time 1 pred time 4 + + Time 1 pred time 5 + + .67 Time 1 pred time 2 + + .59 Time 2 pred time 3 + + .68 Time 3 pred time 4 + + Time 4 pred time 5 + + − .70 WAIS III Symbol Search + .69 .65 − Time 1 pred time 3 + + Time 1 pred time 4 + + .59 .64 Time 1 pred time 5 + + .57 Time 1 pred time 2 + + − Time 2 pred time 3 + + − Time 3 pred time 4 + + Time 4 pred time 5 + + Trail Making Test A .50 + + .57 − + + − + + − .52 + + .46 .51 Page 21 Time 1 pred time 3 Time 1 pred time 4 .50 NIH-PA Author Manuscript NIH-PA Author Manuscript Age + + − Time 1 pred time 2 + + − Time 2 pred time 3 + + Time 3 pred time 4 + + − .69 Time 4 pred time 5 + + − .71 NIH-PA Author Manuscript Time 1 pred time 5 Sex Ethnicity .46 Trail Making Test B .62 + Cysique et al. Overall competence Education Test-retest interval R2 final model Previous score .67 J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Time 1 pred time 3 + + − Time 1 pred time 4 + + − .64 Time 1 pred time 5 + + − .61 + + − + .66 WCST Perseverative errors Time 1 pred time 2 Time 2 pred time 3 + + Time 3 pred time 4 + + + .31 .45 − Time 4 pred time 5 + + Time 1 pred time 3 + + − .51 + .46 + .42 Time 1 pred time 4 + + − .45 Time 1 pred time 5 + + − .30 + + − .49 HVLT-R Total Learning Time 1 pred time 2 Time 2 pred time 3 + + .40 Time 3 pred time 4 + + .46 Time 4 pred time 5 + + Time 1 pred time 3 + + Time 1 pred time 4 + + Time 1 pred time 5 + + Time 1 pred time 2 + + Time 2 pred time 3 + + + + .58 − .42 − .37 .49 HVLT-R Delayed Recall − .43 Page 22 .38 NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Previous score Overall competence + + + + J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Time 1 pred time 3 + + Time 1 pred time 4 + + Time 1 pred time 5 + + Time 1 pred time 2 + + Time 2 pred time 3 + + Time 3 pred time 4 + + Time 4 pred time 5 + + Education Sex Ethnicity Test-retest interval + R2 final model .43 .49 − .39 − Cysique et al. Time 3 pred time 4 Time 4 pred time 5 Age .32 .31 BVMT Total Learning − + .49 .42 .49 + Time 1 pred time 3 + + − Time 1 pred time 4 + + − Time 1 pred time 5 + + + + .45 .45 − + − .49 .46 BVMT Delayed Recall Time 1 pred time 2 − .39 Time 2 pred time 3 + + .29 Time 3 pred time 4 + + .33 − − .33 − .36 − .29 Time 4 pred time 5 + + Time 1 pred time 3 + + Time 1 pred time 4 + + Time 1 pred time 5 + + − .28 + + − .57 − Grooved Peg DH Time 1 pred time 2 Time 2 pred time 3 + + Time 3 pred time 4 + + Time 4 pred time 5 + + Time 1 pred time 3 + + .59 .56 .60 − + + − + + − + .53 .52 Page 23 Time 1 pred time 4 Time 1 pred time 5 .52 NIH-PA Author Manuscript NIH-PA Author Manuscript Age + + − − Education Sex Ethnicity Test-retest interval NIH-PA Author Manuscript Overall competence R2 final model Grooved Peg NDH Time 1 pred time 2 Time 2 pred time 3 + + Time 3 pred time 4 + + J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Time 4 pred time 5 + + Time 1 pred time 3 + + + .56 .61 Cysique et al. Previous score .63 .63 − .57 Time 1 pred time 4 + + − .52 Time 1 pred time 5 + + − .48 pred: predicting Based on effect in overall model: (+) Positive correlation between predictor and outcome; (−) Negative correlation between predictor and outcome. Hierarchical models: order of analysis: 1. Previous test score; 2. overall competence at previous time; 3. age; 4. education; 5. Sex (male 0; female 1); 6 ethnicity (white 1, other 0); 7. Test-retest interval (months) L-N: Letter-Number; WCST-64: Wisconsin Card Sorting Test; HVLT-R: Hopkins Verbal Learning Test-Revised; BVMT: Brief Visuospatial Memory Test-Revised; DH: Dominant hand; NDH, nondominant hand Page 24 Cysique et al. Page 25 Table 6 (Median) practice effect from baseline to follow-up on 15 NP measures (scaled scores) NIH-PA Author Manuscript T2 T3 T4 T5 + Letter fluency 0.0 0.5 1.0 1.0 Animal Fluency 0.0 0.0 0.0 0.0 PASAT 50 0.5 1.0 1.0 1.0 WAIS-III L-N Sequencing 0.0 0.0 0.0 0.0 WAIS-III Digit Symbol 0.0 0 .5 1.0 1.0 WAIS-III Symbol Search 0 .5 1.0 1.0 1.0 Trail Making Test A 0.5 1.0 1.0 1.0 WCST-64 perseverative errors 1.0 2.0 2.0 2.0 Trail Making Test B 1.0 1.0 1.0 1.0 HVLT-R Total Learning 0.0 1.0 0 .5 0 .5 HVLT-R Delayed Recall 0 .5 0 .5 0 .5 0 .5 NIH-PA Author Manuscript BVMT Total Learning 1.0 1.0 0.0 1.0 BVMT Delayed Recall 0 .5 0.0 0.0 0 .5 Grooved Pegboard DH 0.5 0.0 1.0 1.0 Grooved Pegboard NDH 0.0 0 .5 0.5 1.0 Sum 6.0 10.0 10.5 12.5 L-N: Letter-Number; WCST-64: Wisconsin Card Sorting Test; HVLT-R: Hopkins Verbal Learning Test-Revised; BVMT: Brief Visuospatial Memory Test-Revised; DH: Dominant hand; NDH, non-dominant hand NIH-PA Author Manuscript J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 7 J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. T1 T2 T3 T4 T5 T6 2 testings, N=296 9.40 (1.78) - 9.85 (1.87) 9.45 (1.87) - - - - 3 testings, N=241 9.30 (1.81) - 9.76 (1.84) 9.36 (1.84) 9.90 (1.91) 9.24 (1.91) - - - 4 testings, N=171 9.35 (1.77) - 9.81 (1.79) 9.41 (1.79) 9.95 (1.88) 9.28 (1.88) 9.97 (1.96) 9.27 (1.96) - - 5 testings, N=132 9.54 (1.79) - 9.96 (1.77) 9.56 (1.77) 10.12 (1.89) 9.45 (1.89) 10.20 (1.89) 9.50 (1.89) 10.27 (2.00) 9.43 (2.01) - 6 testings, N=64 9.32 (1.81) - 9.74 (1.78) 9.34 (1.78) 9.86 (1.82) 9.20 (1.81) 9.94 (1.76) 9.24 (1.76) 9.98 (1.90) 9.15 (1.91) 10.26 (1.98) 9.36 (1.98) Cysique et al. Uncorrected and practice-effect corrected Mean scaled score across reference samples and across study time points Page 26 NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 8 Tests J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. T1 * T2 N=296 T2 * T3 N=241 T3 * T4 N=171 T4 * T5 N=132 T5*T6 N=64 Mean Scaled Score .91 .91 .92 .91 .88 Letter Fluency .76 .77 .78 .75 .75 Animal Fluency .65 .66 .70 .66 .68 PASAT 50 .77 .82 .85 .84 .82 WAIS-III L-N Sequencing .65 .68 .62 .61 .59 WAIS-III Digit Symbol .82 .84 .89 .91 .92 WAIS-III Symbol Search .73 .78 .82 .79 .82 TMT A .67 .61 .70 .64 .73 TMT B .73 .79 .82 .80 .74 WCST perseverative errors .43 .56 .66 .64 .67 HVLT-R Total Learning .67 .60 .65 69 .53 HVLT-R Delayed Recall .63 .57 .62 .67 .60 BVMT Total Learning .63 .63 .65 .64 .67 BVMT Delayed Recall .55 .48 .52 .49 .52 Grooved Pegboard DH .74 .74 .72 .77 .83 Grooved Pegboard NDH .71 .76 .79 .78 .57 Cysique et al. Reference sample reliability statistics (r) on the 15 NP measures (scaled scores) across study time points. TMT: Trail Making Test; L-N: Letter-Number; WCST: Wisconsin Card Sorting Test; HVLT-R: Hopkins Verbal Learning Test-Revised; BVMT: Brief Visuospatial Memory Test-Revised; DH: Dominant hand; NDH, non-dominant hand. NB: Alternate forms were used at each follow-up sessions for the memory tests: Hopkins Verbal Learning Test-Revised; BVMT: Brief Visuospatial Memory Test-Revised Page 27 NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript Table 9 % decline Times % stable % improve Continuous sRCS J Clin Exp Neuropsychol. Author manuscript; available in PMC 2012 June 1. Reference Sample Validation Sample Reference Sample Validation Sample Reference Sample Validation Sample Reference Sample Validation Sample Time 1 pred Time 2 5% 6% 90% 93% 5% 1% −0.0004 (0.46) −0.16 (0.42) Time 2 pred Time 3 5% 7% 90% 90% 5% 3% 0.0007 (0.47) −0.04 (0.45) Time 1 pred Time 3 5% 6% 90% 91% 5% 3% 0.0000 (0.51) −0.12 (0.45) Cysique et al. Performance of abridged sRCS in the norms development (reference sample versus validation sample) sRCS: summary regression based change score (in these analyses the sRCS is abridged including 8 NP measures: Grooved Peg Board dominant hand and non dominant hand; Trail Making Test A & B; Digit Symbol, PASAT 50; Letter and Semantic Fluency) pred: predicting N=111 (cross validation samples with 2 visits): comparisons with the reference sample showed that the cross-validation sample tended to decline more on the continuous sRCS at time 2. However, there were no significant differences when using the discrete classifications (decline/improve/stable; p=.15). N= 67 (cross-validation samples with 3 visits): comparisons with the reference sample showed that the cross-validation sample tended to decline more on the continuous sRCS continuous sRCS at time 3, but this was not significant (p=.52). Also there were no significant differences when using the discrete classifications (decline/improve/stable; p-values ranging between .22 and .75). Page 28