The amblyopia treatment index

The Amblyopia Treatment Index Stephen R. Cole, PhD,a Roy W. Beck, MD, PhD,b Pamela S. Moke, MSPH,b Marianne P. Celano, PhD,c Carolyn D. Drews, PhD,d Michael X. Repka, MD,e Jonathan M. Holmes, BM, BCh,f Eileen E. Birch, PhD,g Raymond T. Kraker, MSPH,b Kevin E. Kip, PhD,h and the Pediatric Eye Disease Investigator Group Objective: To develop a questionnaire to assess the acceptability of amblyopia treatment and its effect on the child and family. Methods: A 20-item parental survey was developed and pilot tested on 64 subjects, aged 3 to 6 years, participating in the Amblyopia Treatment Study, a randomized trial comparing patching and atropine as treatments for moderate amblyopia. The survey was administered after 4 weeks of treatment. A descriptive item analysis and an internal consistency reliability analysis were performed. Results: Nineteen of the 20 items demonstrated adequate variability as evidenced by the frequency distributions for item responses. Only 4 (<1%) of 1280 possible item responses were missing, one each by 4 different respondents. Factor analysis identified 3 treatment-related factors—”adverse effects,” “compliance,” and “social stigma”—among 11 of the 20 items. The internal-consistency reliability α for the 5-item adverse effects subscale was 0.82, the 4-item compliance subscale α was 0.81, and the 2-item social stigma subscale α was 0.84. Conclusions: The Amblyopia Treatment Index appears to be a useful instrument for assessing the impact of amblyopia treatment in 3- to 6-year-old children. (J AAPOS 2001;5:250-4) I n the Amblyopia Treatment Study, we are comparing patching and atropine as treatments for moderate amblyopia in 3- to 6-year-old children. To complement the primary study endpoint, ie, improvement in visual acuity, we developed the Amblyopia Treatment Index (ATI) as a survey instrument to assess the acceptability of treatment and its impact on the child and family. This measure will be of particular importance in the trial if the visual acuity improvement is equivalent in the 2 treatment groups, as hypothesized. Herein, we report the results of an initial pilot study conducted to evaluate the relative ease of administration of the questionnaire and to assess some of its psychometric properties. METHODS The process of questionnaire development is a well-developed field of study with many concepts arising out of psychometrics.1 With a specific objective in mind, a typical first From the Department of Epidemiology, Johns Hopkins School of Public Health, Baltimore, Marylanda; the Jaeb Center for Health Research, Tampa, Floridab; the Department of Psychiatry and Behavioral Sciences, Emory University School of Medicinec and the Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgiad; the Wilmer Ophthalmological Institute, Johns Hopkins University School of Medicine, Baltimore, Marylande; the Mayo Clinic, Rochester, Minnesotaf; the Retina Foundation of the Southwest, Dallas, Texasg; and the University of South Florida, Tampa, Florida.h Supported by a cooperative agreement from the National Eye Institute EY11751. Submitted February 16, 2001. Revisions accepted May 7, 2001. Reprint requests: PEDIG Data Coordinating Center, Jaeb Center for Health Research, 3010 E 138th Ave, Suite 9, Tampa, FL 33613. Copyright © 2001 by the American Association for Pediatric Ophthalmology and Strabismus. 1091-8531/2001/$35.00 + 0 75/1/117097 doi:10.1067/mpa.2001.117097 250 August 2001 stage of questionnaire development is to canvass subject-area specialists to elicit content areas and build a pool of possible items. After refinement and culling (possibly by an informed committee), this pool of items is pilot tested on a representative group of subjects. By following these principles, an initial listing of content areas to be covered by the items in the questionnaire was developed from knowledge of the literature and clinical experience. Items were designed to tap several aspects of impact, including parenting stress, concern about how others may perceive the child, strained family relationships, difficulty with treatment adherence, etc. The items were created with the intent of completion by a parent or guardian, with the content geared toward 3- to 6-year-old children, the age range of the patients in the Amblyopia Treatment Study. The initial list of items was reviewed for content validity by several pediatric eye care specialists, a pediatric psychologist, and parents of children with amblyopia, and then reduced in number to 20 items (Table 1). These 20 items were chosen with the intent of creating independent subscales assessing several dimensions of the impact of treatment, such as effect, adverse events, and compliance. The measure consists of 20 Likert-type items with 6 response choices ranging from “strongly agree” to “strongly disagree,” with higher scores (ie, “disagree”) indicating more (adverse) impact. Several items were reversed to minimize response biases. While a single item set was being developed, the wording was adjusted slightly to reflect each treatment protocol. A write-in section was included asking the parent to provide comments not covered in the questionnaire. To accommodate the time line for the treatment trial, the ATI was implemented in the trial with a plan to assess its performance after a minimum of 40 patients. Journal of AAPOS Journal of AAPOS Volume 5 Number 5 August 2001 Cole et al 251 TABLE 1. The ATI* Patching questionnaire 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 My child does not seem to mind wearing the patch once it is on. I worry that by wearing the patch, my child may miss out on fun activities (such as games and parties). Wearing the patch affects my child’s learning. Wearing the patch makes it hard for my child to play outside, such as running, jumping, or riding a bike or tricycle. I have trouble putting on my child’s patch and keeping it on. Wearing the patch is a source of tension or conflict in my relationship: a. with my child. b. with another family member. c. with my child’s babysitter or teacher. Wearing the patch makes it difficult for my child to draw, color, or write. I worry that my child will become injured when wearing the patch. My child can see well when wearing the patch. My child complains when it is time to wear the patch. Wearing the patch makes my child’s eye or eyelids red or irritated. I worry that my child does not wear the patch enough. My child is more clumsy and uncoordinated than usual when wearing the patch. I notice that other children stare at my child when the patch is on. I believe that wearing the patch will improve my child’s vision. Wearing the patch makes it difficult for my child to play with blocks or toys. I sometimes forget to put the patch on my child. I worry that wearing the patch will make my child feel different from other children. Atropine questionnaire My child does not seem to mind using the drops. I worry that by using the drops, my child may miss out on fun activities (such as games and parties). Using the drops affects my child’s learning. Using the drops makes it hard for my child to play outside, such as running, jumping, or riding a bike or tricycle. I have trouble putting the drops in my child’s eye. Using the drops is a source of tension or conflict in my relationship: a. with my child. b. with another family member. c. with my child’s babysitter or teacher. Using the drops makes it difficult for my child to draw, color, or write. I worry that my child will become injured when using the drops. My child can see well when using the drops. My child complains when it is time to put in the drops. Using the drops makes my child’s eye or eyelids red or irritated. I worry that my child does not get the drops often enough. My child is more clumsy and uncoordinated than usual when using the drops. I notice that other children stare at my child when the drops are in. I believe that using the drops will improve my child’s vision. Using the drops makes it difficult for my child to play with blocks or toys. I sometimes forget to put the drops in my child’s eye. I worry that using the drops will make my child feel different from other children. *In the questionnaire, each item has 6 response choices: strongly agree, agree, neither agree or disagree, disagree, strongly disagree, and not applicable. Patients in the Amblyopia Treatment Study must be younger than 7 years and mature enough to complete the study’s visual acuity testing protocol (which creates an effective lower age limit of about 3 years old).2 Previous amblyopia therapy could be no more than 2 months in the prior 2 years. Each patient is randomly assigned to treatment with either patching (initially 6 to 12 hours per day at investigator discretion) or atropine 1% (1 drop per day). The ATI was completed by the parent or guardian accompanying the child to the office at the time of the first follow-up visit (after 4 weeks of treatment). Consent for its completion was included in the overall study informed consent form. The questionnaire was not completed if an individual responsible for administering the treatment did not accompany the child to the 4-week visit. At the clinic, before the child’s examination, the parent was given brief verbal instructions and then completed the questionnaire, which he or she sealed in a preaddressed postage-paid envelope that was mailed to the Data Coordinating Center. Two essential psychometric properties are often quantified at the pilot-testing stage, namely, reliability and validity. Reliability may be viewed as the repeatability of test items and is often measured by either a Pearson correlation coefficient on test-retest administrations or a statistic called Cronbach α on a single test administration.3 A desirably high reliability (>0.8) suggests that subjects given the questionnaire on 2 different occasions (close enough in time such that the underlying process being measured has not changed) would score similarly. Broadly defined, validity is the property of measuring what one intends to measure. Validity may be measured externally by correlating questionnaire results to results on preexisting questionnaires that should be either positively or negatively associated with the novel questionnaire. Alternatively, internal validity may be measured by examining the factor structure of the novel questionnaire: namely, how the individual items correlate with underlying factors (items shared variance) in the questionnaire.1,4 High internal validity is the result of a simple factor structure with high correlations (>0.5) between individual items and the underlying factors. Also, the underlying factors should mirror the concepts intended to be measured. Of course, internal validity is of utmost importance when preexisting questionnaires are scarce. By using this approach, analyses were conducted after the ATI had been completed for 64 study patients; data from both treatment groups were combined for analysis. With a sample of 64 and an assumed internal consistency reliability coefficient of 0.70, the 2-sided 95% CI for Cronbach α would range from 0.56 to 0.79, demonstrating adequate precision. Descriptive item analyses and internal consistency reliability analysis were performed. Frequency tables 252 Journal of AAPOS Volume 5 Number 5 August 2001 Cole et al TABLE 2. Amblyopia treatment index item responses (n = 64)* Item (abbreviated)† 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Child does not seem to mind treatment Worry child may miss out on fun activities Treatment affects child’s learning Treatment makes it hard for child to play Trouble applying treatment to child Treatment is a source of tension for me: a. with child b. with another family member c. with my child’s babysitter or teacher Difficult for child to draw, color, or write Worry child will be injured on treatment Child can see well on treatment Child complains when it is time for treatment Child’s eye or eyelids become red or irritated Worry that child not getting enough treatment Child clumsy on treatment Other children stare at child Treatment will improve child’s vision Treatment makes it difficult for child to play Sometimes forget to apply treatment to child Worry that child feels different Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Does not apply 24 0 0 1 2 25 7 4 7 5 5 4 8 3 6 4 22 23 25 21 2 27 25 24 26 0 0 0 0 0 2 1 0 0 0 6 5 1 3 0 3 26 0 3 1 1 1 1 5 6 24 16 7 3 9 17 31 1 13 12 2 1 0 9 10 17 7 5 6 14 7 3 9 2 7 25 19 19 23 21 11 18 20 20 19 12 0 24 17 19 30 34 30 23 23 2 14 27 28 18 21 0 26 25 21 0 4 10 0 0 0 0 0 0 0 0 0 0 0 0 *Sixty of 64 respondents completed all items. †See Table 1 for complete wording of each item. were used to describe the item-response distributions to assess whether any items had limited range. Next, exploratory factor analysis was performed to investigate the underlying factor structure of the ATI. Any factor with an eigenvalue (the amount of combined item variance accounted for by the factor) greater than 1 was retained for interpretation.4 Item loadings, an estimate of the correlation between the item and the underlying factor, greater than 0.5 were considered noteworthy. Items that did not load strongly on any factor, that loaded on several factors equally, or whose removal would increase the internal consistency reliability were not included in the final exploratory factor analysis nor included on subscales. Subscales were created by using a simple, equally weighted, summated rating scale for the items that appeared to load on a similar factor. Finally, the internal-consistency reliability for the ATI subscales was estimated by Cronbach α.3 Subscale development analyses were limited to the 60 subjects with all ATI items present. There were no obvious differences when the analyses were conducted stratified by treatment group in this small sample (data not shown). RESULTS The 64 study patients had an average age of 5.0 ± 1.1 years; 39% were female. Sixteen (25%) had been treated previously for amblyopia, usually (14 of 16) with patching. The mother was the respondent for 80% (n = 51), the father for 19% (n = 12), and guardian for 1% (n = 1) of the questionnaires that were completed. Ninety-nine percent of the respondents indicated that they were the individual who was responsible for the treatment all or most of the time. Missing responses to the ATI items were minimal. Only 4 (<1%) of 1280 possible item responses were missing, one each by 4 different respondents. Nineteen of the 20 items demonstrated adequate variability as evidenced by the frequency distributions for item responses (Table 2). One item (No. 15: “I believe that treatment will improve my child’s vision”) was limited in the response range with 95% of the responses congregated in the “strongly agree” or “agree” category. From exploratory factor analysis, 12 of the 20 items were correlated strongly (>0.50) with 1 of 3 underlying factors that had an eigenvalue greater than 1. A 3-factor solution was also suggested by visual inspection of a scree plot. After an iterative process of removing items that did not load strongly on any factor (items 2, 6c, 8, 11, and 17), that loaded equally on more than 1 factor (items 4, 6a, and 6b), or whose removal increased the scale reliability (item 15), we termed the 3 treatment-related factors adverse effects, compliance, and social stigma (Table 3). These 3 factors accounted for nearly 100% of the shared variance among the 11 analyzed items (or 69% of the total variance from a principal components analysis).4 By using an oblique rotation, which allows the 3 factors to be correlated, the correlation between adverse effects and social stigma was 0.42; between adverse effects and compliance was 0.44; and between social stigma and compliance was 0.42. Journal of AAPOS Volume 5 Number 5 August 2001 Cole et al 253 TABLE 3. Correlation between items and factors from exploratory factor analysis (n = 60)* Factor loadings for retained items 3 7 9 13 16 1 5 10 12 14 18 Treatment affects child’s learning Difficult for my child to draw, color, or write Child can see well on treatment Child clumsy on treatment Treatment makes it difficult for child to play Child does not seem to mind treatment Trouble applying treatment to child Child complains when it is time for treatment Worry child not getting enough treatment Other children stare at child Worry that child feels different Adverse effects of treatment Treatment compliance Social stigma 0.69† 0.83 0.52 0.58 0.70 0.23 0.21 0.04 0.22 0.06 0.18 0.08 0.19 0.11 0.18 0.29 0.79 0.56 0.78 0.61 0.10 0.24 0.05 0.10 0.02 0.47 0.28 0.05 0.15 0.11 0.37 0.80 0.79 *Items 2; 4; 6a, b, and c; 8; 11; and 17 did not load strongly (>0.50) on any single factor. See Table 1 for full wording of each item. †Estimated factor loadings > 0.5 are in bold. The internal-consistency reliability for the 5-item adverse effects subscale was 0.82, for the 4-item compliance subscale α was 0.81, and for the 2-item “social stigma” subscale α was 0.84. DISCUSSION The ATI was developed to provide a measure of the impact of amblyopia therapy on the child and parent. In a pilot study conducted with 64 patients enrolled in the Amblyopia Treatment Trial, we found the ATI to be readily self-administered by parents, with limited verbal instructions required. Missing data were minimal. Three distinct subscales were observed after using exploratory factor analysis: a 5-item adverse effect scale, a 4-item compliance subscale, and a 2-item social stigma subscale. Each of these subscales appeared to have strong content validity and had an internal consistency reliability estimate greater than 0.80. The questionnaire was completed by a parent or guardian after 4 weeks of treatment. This was considered a sufficient length of time to be able to evaluate the impact of treatment on the child and parent. This time point coincided with the first follow-up visit so that the questionnaire was completed before the child’s visual acuity was tested, therefore minimizing the impact of knowledge of the child’s improvement from influencing the questionnaire response. Although a longer treatment period before completion would have provided broader information on the impact of treatment, it would also potentially bias the results because knowledge of visual improvement at the follow-up examinations could influence the responses. This would make the questionnaire results difficult to interpret, particularly if there was a difference between groups in visual recovery. In general, the content of the questions appeared appropriate and comprehensive. The open-ended question, which requested information on areas not covered by the items, did not elicit any responses that suggested that the wording of existing items should be modified or that new items should be added. One item (No. 15 regarding the parent’s expectation of visual improvement from treatment) had a range of responses that were too narrow for it to be useful. This likely reflects the fact that the parent’s expectation had already been strongly influenced by the information, which was given at the time of informed consent for the study, that both treatments were expected to improve vision. Parent’s expectations might also have been influenced by visual improvement that may have occurred during the first month of treatment. This question would be more properly asked before the start of treatment. The 3-factor solution identified herein should be interpreted in the context of methodologic limitations. First, the sample of 64 patients is less than the minimum-recommended guideline of the larger 100 subjects, or 5 times the number of variables (items) being analyzed.4 In addition, results from the oblique rotation suggested that the underlying factors—adverse effects, social stigma, and compliance— may be modestly correlated, rather than truly orthogonal (independent). Finally, the social stigma subscale was formed on the basis of only 2 items, which is less than we would recommend. Acknowledging these limitations, and the exploratory nature of this analysis, we have elected not to make any changes in the ATI for its use in the Amblyopia Treatment Study. The questionnaire data will be fully analyzed after completion of the 400-patient trial. If the results are similar to the current results, we may recommend a shortened version of the ATI. The questionnaire is geared toward children in the 3to 6-year-old age range; therefore, certain questions might not apply to younger or older children. If the ATI is used strictly for assessing the impact of patching, it would be advisable to split question 5 into two questions because it covers two concepts—applying the patch and keeping the patch on. For our purposes, this item was written as such to have a single question related to the treat- 254 Journal of AAPOS Volume 5 Number 5 August 2001 Cole et al ment application for both the atropine and patching groups. In summary, the ATI appears to be a useful instrument for assessing the impact of amblyopia treatment on 3- to 6year-old children and their parents. We expect that this questionnaire will be useful to other researchers conducting studies of amblyopia treatment. O References 1. Nunnally JC, Bernstein IH. Psychometric theory. New York: McGraw-Hill; 1994. 2. Holmes JM, Beck RW, Repka MX, et al. The Amblyopia Treatment Study visual acuity testing protocol. Arch Ophthalmol; in press. 3. Bland JM, Altman DG. Cronbach’s alpha. BMJ 1997;314:572. 4. Hatcher L. Using the SAS system for factor analysis and structural equation modeling. Cary (NC): SAS Institute; 1994. N THE MOVE? Send us your new address at least six weeks ahead Don’t miss a single issue of the journal! To ensure prompt service when you change your address, please photocopy and complete the form below. Please send your change of address notification at least six weeks before your move to ensure continued service. We regret we cannot guarantee replacement of issues missed due to late notification. JOURNAL TITLE: Fill in the title of the journal here. OLD ADDRESS: Affix the address label from a recent issue of the journal here. NEW ADDRESS: Clearly print your new address here. Name Address City/State/ZIP COPY AND MAIL THIS FORM TO: Mosby Subscription Customer Service 6277 Sea Harbor Dr Orlando, FL 32887 OR FAX TO: 407-363-9661 OR PHONE: 1-800-654-2452 Outside the US, call 407-345-4000