Research in Developmental Disabilities 35 (2014) 2329–2340

Contents lists available at ScienceDirect

Research in Developmental Disabilities

Caregiver’s Concerns-Quality of Life Scale (CC-QoLS):

Development and evaluation of psychometric properties
Gemma Unwin a, Shoumitro Deb b,*
a
School of Psychology, University of Birmingham, UK
b
Imperial College London, Department of Medicine, Division of Brain Sciences, UK

A R T I C L E I N F O A B S T R A C T

Article history: We have developed a Caregiver’s Concerns-Quality of Life Scale (CC-QoLS) for adults with
Received 19 May 2014 intellectual disabilities (ID) who exhibit aggressive behaviour. The CC-QoLS is a brief (8
Accepted 22 May 2014 items in each subscale, CC and QoL respectively) proxy measure completed by caregivers.
Available online 19 June 2014
This is a specific health related quality of life instrument (HRQoL) combined with
measures of caregiver’s concerns for use as an outcome measure to assess clinical and cost
Keywords:
effectiveness of interventions for aggression in adults with ID. The CC-QoLS was found to
Intellectual disabilities
have good face validity and very good test–retest reliability with an ICC of 0.81 for CC
Adults
Health related quality of life scale (range 0.46–0.83 across items) and 0.80 for QoL (range 0.65–0.81 across items). Similarly,
Caregiver’s concerns scale the scale had good inter-rater reliability with an ICC of 0.67 for CC (range 0.31–0.63 across
CC-QoLS items) and 0.63 for QoL (range 0.31–0.65 across items). Internal consistency for each
Psychometric properties subscale was also good (Cronbach’s alpha was 0.85 for CC and 0.80 for QoL; Split-half
Spearman–Brown was 0.81 for CC and 0.70 for QoL). Furthermore, the scale showed good
concurrent validity with measures of severity of aggressive behaviour, namely Modified
Overt Aggression Scale (MOAS) (CC: r = 0.4; p  0.01 and QoL: r = 0.2; p  0.05) and
Aberrant Behavior Checklist-Irritability subscale (ABC-I) (CC: r = 0.5; p  0.01 and QoL:
r = 0.02; p  0.05) as well as Caregiver’s Uplift and Burden Scale score (<0.05). We believe
that the CC-QoLS is a user friendly, easy to complete, first-ever HRQoL measure for adults
with ID and aggressive behaviour with very good psychometric properties.
ß 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Quality of life (QoL) is a multidimensional construct recognised as an essential component in outcome measurement
(Bertelli & Brown, 2006; Bertelli et al., 2013; Townsend-White, Pham, & Vassos, 2012) in the field of intellectual disabilities
(ID). As Verdugo, Schalock, Keith, and Stancliffe (2005) suggest, ‘the importance of the valid assessment of QoL is heightened
with the increased use of the QoL construct as a basis for policies and practices in the field of ID’ (p. 707). Many QoL scales for
the general non-ID population are intended to be completed by the person themselves, to measure subjective QoL. However,
this poses a problem in ID research where informants are often used in data collection owing to communication and
cognitive difficulties of the person themselves. In clinical and research settings, sometimes the person themselves may not
be able to express how they feel and whether they feel any better, and often such information is collected from a caregiver
attending the clinic to support the person. One problem with informant-rated proxy QoL scales is that it could be argued that

* Corresponding author.
E-mail addresses: g.l.unwin@bham.ac.uk (G. Unwin), s.deb@imperial.ac.uk (S. Deb).

http://dx.doi.org/10.1016/j.ridd.2014.05.018
0891-4222/ß 2014 Elsevier Ltd. All rights reserved.
2330 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

informants may not and cannot reliably report on the subjective QoL of someone else such as commenting on life satisfaction
experienced. Therefore, informant-rated scales tend to use more objective indicators of QoL. It would be ideal, if possible, to
have both subjective as well as objective components in a QoL scale (Verdugo et al., 2005). However, this may not be feasible
in practice for some people with ID, particularly for those with severe and profound ID who have limited or no expressive
verbal or written communication. Because of this Verdugo, Gómez, Arias, Navas, and Schalock (2014) have recently
developed an informant based QoL measure.
QoL scales designed for the general non-ID population often have objective items which may not be relevant to an
individual with ID who, because of their impairments, may not be able to go out on their own, live on their own, manage their
money or look after a family. Therefore, specific QoL scales are required for people with ID. However, a recent systematic
review of QoL measures for people with ID could not locate any instruments that assessed QoL specifically among people
with ID who also show problem behaviour (Townsend-White et al., 2012). The authors conclude that ‘more instruments that
measure [QoL] need to be developed and rigorously validated’ (p. 270).
It is important to measure QoL when evaluating the effectiveness of interventions. In the past, studies which have
investigated the effects of interventions on aggressive behaviour have often used outcome measures which addressed
severity and frequency of the behaviour but which did not assess the impact on QoL for the individual with ID. However, it is
known that interventions may be very effective at reducing the severity and frequency of a problem behaviour but may also
have adverse effects which reduce QoL. Therefore, a more holistic assessment of outcomes for the individual, which includes
a QoL assessment, can provide more accurate information about the utility of an intervention.
Townsend-White et al.’s (2012) recent systematic review identified only six reasonable quality QoL measures for use in
people with ID which used a wide variety and number of items. Most scales aim to measure subjective QoL from the person
concerned and only a handful can be used to measure QoL via a proxy who answers on behalf of the person. Furthermore, few
had been robustly evaluated in relation to their psychometric properties.
The health related QoL (HRQoL) measures are different from the generic QoL measures in that they are based on the
concept of impairment vs. activities (previously called disabilities) vs. participation (previously called handicap) as
described in International Classification of Functioning (ICF) (World Health Organization, 1980, 1997), whereas the generic
QoL measures cover very broad concepts. Similarly the HRQoLs are different from other health outcome measures which are
based on symptom severity that put emphasis on the impairment part of ICF classification rather than activities and
participation. This notion of participation based outcome measure is evident in most widely used HRQoLs in health care
worldwide such as EQ-5D (EuroQoL Group, 1990), SF-12 (Brazier, Roberts, & Deverill, 2002), and World Health Organization
Disability Assessment Schedule (WHODAS 2.0) (World Health Organization, 2012). Therefore, these scales measure the
impact of symptoms such as walking difficulty on the daily living skills and performances. Although this approach is
common and widely used in generic health care, this has hardly been used in relation to health outcome measures in people
with ID, an approach we specifically wanted to adopt in developing our scale.
The HRQoLs that are commonly used for non-ID general population such as EQ-5D and SF-12 are not valid for use in the ID
population. Therefore, there is an urgent need for development of specific HRQoLs for the ID population, with sound
psychometric properties that could be used in the evaluation of cost effectiveness of a particular intervention.
Although QoL and caregiver concern are two important aspects of outcome, these are seldom included in pharmacological
intervention studies in the field (Bertelli et al., 2013; Hemmings, Deb, Chaplin, Hardy, & Mukherjee, 2013). In order to
address this deficiency, we developed and field tested a mental health related QoL (HRQoL) measure along with a caregiver
concern scale for adults with ID who show problem behaviours, particularly aggression. This was done in the context of a
larger observational study; Naturalistic Observational Study of Different Interventions for aggression in adults with ID
(NOSDIID). The NOSDIID study (Unwin & Deb, 2009) evaluated outcomes of different interventions and trajectory of
aggressive behaviour in adults with ID over 12 months using a variety of outcome measures. A Caregiver’s Concerns-Quality
of Life Scale (CC-QoLS) was developed and used as one of the outcome measures.

2. Methods

2.1. Development of the Caregiver’s Concerns-Quality of Life Scale (CC-QoLS)

In order to develop the CC-QoLS, we followed the steps described by Streiner and Norman (2003) for developing a new
health measurement tool. These methods are widely accepted as gold standards for developing a new instrument. One
author (SD) who is a Consultant Psychiatrist and a Clinical Professor specialising in ID in the UK originally developed the scale
for use in his psychiatric clinics with the intention of more objectively measuring intervention effects in relation to QoL of
the patients and their caregivers rather than subjectively assessing the outcome from caregivers’ retrospective reporting. The
scale was developed due to a lack of appropriate available scales that were quick and easy to use in the clinic and had proven
reliability and validity. The aim was to develop a brief, informant-rated, holistic measure to assess outcome in adults with ID
who exhibit aggressive behaviour.
Opinions were collected from adults with ID and their caregivers who attended for the management of problem
behaviour and epilepsy in SD’s psychiatric clinic in the West Midlands region of the UK. In the clinic people with ID were
asked about their expectations from the treatment apart from symptom reduction. They were asked to comment on the ways
they thought that treatment may eventually help them to improve their quality of life. Caregivers were also asked which
 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340 2331

areas of patients’ lives they felt were important in terms of outcome. Both patients and their caregivers suggested a number
of areas which were important to them in the day to day life of patients where improvement could be made as a result of the
treatment. These were compared against the observation of patient’s behaviours and caregivers’ reporting in the follow up
clinics.
Over a period of time it became apparent that despite some overlap, the areas patients chose as important for their quality
of life in terms of treatment outcome were not necessarily the same as the ones caregivers chose as their own areas of
concern in terms of expectation from treatment intervention. For example, family caregivers consistently expressed concern
about patients’ overall safety (‘risk of accident’) and lack of appropriate service provision. On the other hand, the patients
themselves felt that ‘relationship with others’ is an important factor which affected their QoL in relation to intervention for
aggressive behaviour. For example, some said that having friends both within the group home and outside (‘relationships
with others’) is important as they may be losing friends because of their aggressive behaviour. Some felt that at times they
became frustrated because of lack of friends or support in the house which led to aggression. Some reported that sometimes
negative relationships with other residents in the group home or members of staff could lead to aggression. Similarly, many
adults with ID felt that ‘communication skills’ (as opposed to speech) both within the home and outside is very important to
them as they felt that others do not understand them properly and it is this lack of understanding and communication that
sometimes leads to frustration and aggression.
Themes gathered in the clinic were compared with the items in QoL measures available in the literature and other
outcome measures of health interventions for people with ID. We modelled our scale in line with the commonly used
HRQoLs in the non-ID population such as EQ-5D, SF-12 and WHODAS 2.0 (discussed in the Introduction section).
Eventually 16 items were included in the CC-QoLs which were felt to have broadly reflected views of caregivers, adults
with ID, the clinicians in the field and what was already available in the literature. The CC-QoLS was then used in SD’s
psychiatric clinic to assess the feasibility of its use on the ground such as whether the caregivers had any difficulty in
completing the questionnaire, whether the wording was unambiguous etc. Caregivers found the CC-QoLS easy to
complete usually within 5 minutes, which is useful in a clinic setting. The CC-QoLS was then field tested during the
NOSDIID study.
The CC-QoLS comprises two parts to measure two constructs, namely outcomes for caregivers, in relation to concerns
they have about the person they care for (CC) as well as QoL for the adult with ID. The Caregiver Concerns (CC) subscale and
QoL subscale each comprises 8 items respectively. A five-point rating scale was used to assess each item (see Appendix A).
Each item (for example, ‘physical health’) of the CC subscale was rated as ‘not concerned’ = ‘0’; ‘mildly concerned’ = ‘1’;
‘moderately concerned’ = ‘2’; ‘very concerned’ = ‘3’; and ‘seriously concerned’ = ‘4’. Each QoL item (for example, ‘relationships
with others’) was measured as ‘very poor’ = ‘0’; ‘poor’ = ‘1’; ‘satisfactory’ = ‘2’; ‘good’ = ‘3’; and ‘very good’ = ‘4’. The range for
each subscale score is therefore 0–32. The scores of CC and QoL could not be combined as they are rated in opposite direction
in that a higher CC score means higher concern and a higher QoL score means a better QoL.
Commonly, five-point scales are advised in scale development literature (Clark-Carter, 2001). As Lissitz and Green (1975)
suggest, there is a levelling off in increased reliability after five scale points. In addition, Cummins (1997), in a review of QoL
measures for people with ID, suggests that five-point scales generally reflect about 95% of the underlying variation.
Cummings also highlights that there is a trade-off between increased reliability when there is a binary choice and issues of
discrimination and validity which require a larger number of choice points (see Chang, 1994 for a review of these issues).

2.2. The NOSDIID study

The psychometric properties of the CC-QoLS had not yet been established. Therefore, through the course of the NOSDIID
study, data were collected to allow for the assessment of various aspects of reliability and validity of the scale. In the NOSDIID
study, data were collected on different aspects of participant characteristics, psychiatric and behavioural measures along
with caregiver uplift and burden on 100 adults (aged 18 years or over) with ID (mild to profound) who attended different ID
psychiatric clinics in the West Midlands region of the UK for the management of aggressive behaviour. One author (GU)
collected information on various assessment scales by interviewing either a paid or a non-paid family caregiver who knew
the person well. All measures, along with the CC-QoLS, were completed at baseline (T1) and after 6 months (T2) and 12
months (T3).
The study received approval from the local medical research ethics committee in the UK and GU collected informed
consent from the person with ID and her/his caregiver. Where the person with ID did not have capacity to give consent an
assent was agreed with a ‘Consultee’ as per the UK regulation on research involving people who are unable to give an
informed consent.

2.3. Other outcome measures used in the NOSDIID study

In addition to CC-QoLS, a number of other outcome measures were used in the NOSDIID study to map the trajectory of
aggressive behaviour over time and the effect of different interventions. Aggressive behaviour was measured using Modified
Overt Aggression Scale (MOAS) (Ratey & Gutheil, 1991). Broader problem behaviours were measured using Aberrant
Behaviour Checklist-Irritability subscale (ABC-I, Aman, Burrow, & Wolford, 1995), and caregiver burden and uplift was
measured using Caregiver Uplift and Burden Scale (Pruchno, 1990).
2332 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

The MOAS (Ratey & Gutheil, 1991) is a modified version of the Overt Aggression Scale (OAS) (Silver & Yudofsky, 1991),
which was developed to document aggressive episodes when they occur and to assess the effectiveness of interventions in
the management of violent patients. The MOAS has been validated to be a practical and effective measure of aggression and
has been used successfully in people with ID (Oliver, Crawford, Roa, Reece, & Tyrer, 2007; Ratey & Gutheil, 1991; Tyrer et al.,
2008). The MOAS has 16 classes of aggressive behaviour grouped into four subscales (verbal aggression, physical aggression
against objects, physical aggression against the self and physical aggression against others). Each type of aggressive
behaviour is rated by informants on a 5-point scale of increasing severity, from 0 for absent to 4, indicating the most severe
type of this behaviour.
We have used weighted scores on the MOAS subscales (through multiplication) to provide a more accurate measure of
severity (Crocker et al., 2006). To calculate the weighted MOAS score, the score for verbal aggression is multiplied by one,
aggression against property is multiplied by two, aggression against self is multiplied by three, and aggression to other
people is multiplied by four, the scores are then totalled, to give a possible score of 0–40 (Crocker et al., 2006).
The ABC-Community (ABC-C, Aman et al., 1995) is a widely used scale with well-established psychometric properties and
consistently reported validity and reliability (Unwin & Deb, 2008). It measures a wide range of problem behaviours over the
month preceding assessment. It has been shown to be a meaningful measure of intervention effects and initial assessments
of people with ID and co-morbid mental illness (Shedlack, Hennen, Magee, & Cheron, 2005) as well as for assessing problem
behaviour in its own right. The ABC-C has 58 behavioural items, rated by informants on a 4-point scale from ‘0 = not a
problem’ to a ‘3 = severe problem’. The ABC-C has five empirically derived factors with the irritability, agitation and crying
subscale (ABC-I) closely corresponding with the present definition of aggressive behaviour and incorporating similar
behaviour as outlined in the MOAS. We have decided to use ABC-I in this study as the total score of ABC is not valid (Aman
et al., 1995) and most intervention studies now use ABC-I as the primary outcome for aggression in ID in randomised
controlled trials (Deb, 2013).
The Carer’s Uplift and Burden Scale (Pruchno, 1990) was originally developed to measure perceived carer uplift and
burden in spouse caregivers of people with mental health problems. Whilst not originally designed for caregivers of people
with ID, it has recently been successfully used with this population in a randomised controlled trial and is suitable in the
present context as it is specifically designed for informants (Tyrer et al., 2009). The questionnaire asks family caregivers to
rate six uplift items and 17 burden items for the last four weeks on a three-point Likert scale from ‘not at all/never’ to ‘most of
the time/often’. The scores for each construct are aggregated to provide a possible score range of 6–18 for uplift and 17–51 for
burden. The uplift items reflect the daily ‘pleasures or compensations afforded by life as a caregiver’ (Pruchno, 1990, p. 62).
Items on the burden scale reflect the more negative psychological aspects of caring. Internal consistency for each construct is
high (coefficient alpha = 0.8 for uplift and0.9 for burden (Pruchno, 1990)).

2.4. Assessment of psychometric properties

The assessments of reliability and validity followed the methods outlined by Streiner and Norman (2003). The CC-QoLS
was initially administered in a pilot phase during three interviews with caregivers to assess face validity. The interviewer
(GU) obtained verbal feedback from caregivers to assess how well the questionnaire was received and made notes based on
their impressions of the utility and feasibility of the scale.
Data were collected at T1 to allow for test–retest and inter-rater reliability testing. Test–retest seeks to investigate the
consistency of measurement over time, therefore, caregivers were asked to complete two copies of the questionnaire so that
the correlation between their scores could be investigated. At the first interview, caregivers completed the CC-QoLS in the
presence of the researcher (GU) who then left a blank copy with a freepost envelope to be completed in a week’s time and
posted back. Blinding to previous response was ensured as the initially completed copy of the CC-QoLS was kept by the
researcher (GU) after the interview. The results of the two administrations were analysed using intra-class correlation
coefficients (ICC).
Where available at T1, another caregiver was approached to complete an additional copy of the CC-QoLS, at the same
time, but, in isolation to the original caregiver to allow for the assessment of the consistency of the measure across raters
(inter-rater reliability). Ideally, raters would have been chosen at random, however, they were sampled on an opportunity
basis whereby another family or staff member of the same household was asked to complete an additional copy of the CC-
QoLS.
ICC was used for test re-test and inter-rater reliability analyses as it provides a reliability coefficient based on ‘the
correlation between one measurement (either a single rating or a mean of several ratings) on a target and another
measurement obtained on that target’ (Shrout & Fleiss, 1979, p. 422). In this way, ICC is suitable for estimating both inter-
rater and test–retest reliability. Furthermore, ICC is commonly quoted as the preferred statistic for estimating inter-rater
reliability (Streiner & Norman, 2003).
Cronbach’s alpha was used to model internal consistency of each subscale by averaging the correlation among items.
Furthermore, a correlation matrix was computed to show the correlations between each of the items on each scale. These
correlations indicate whether the scale is measuring a single dimension. Items which inter-relate well suggest that the items
are related and therefore measuring the same underlying dimension. Similarly, the item-total correlations were inspected to
ensure each item was adequately correlated with the subscale total score. Split-half reliability using Spearman–Brown
correlation coefficient also assessed the homogeneity of each sub-scale (Streiner & Norman, 2003). Furthermore, the
 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340 2333

two-factor structure of the scale was assessed using bivariate Spearman’s correlation for the total subscale scores to
investigate the validity of the separation of the subscales of the questionnaire and the extent to which the total score for each
subscale were related.
Criteria-related validity was measured by assessing concurrent validity by correlating (Spearman’s) CC-QoLS score with
other outcome measures used in the NOSDIID study such as weighted MOAS score, ABC-I score, family caregiver’s uplift score
and family caregiver’s burden scores at T1. Contamination of scores was prevented by the caregivers being unaware that
their scores on one scale would be assessed against their scores on another. One way repeated measures ANOVA and
Friedman’s ANOVA were used to assess whether the each subscale of the CC-QoLS varied over time (QoL and CC respectively).
All data were entered into the Statistical Package for Social Sciences (SPSS; IBM, version 21) to allow for statistical
analysis.

3. Results

Data on the CC-QoLs were available from 99 caregivers at T1. Table 1 provides a summary of the scores for each subscale
and item on the CC-QoLS.

3.1. Face validity

Caregivers reported that the questionnaire was easy to understand and the items were relatively simple. They felt that
each item was relevant and important to measure. They felt that the scoring method was appropriate and provided enough
options without being too detailed, meaning that it was not difficult to select an option. The range in scores, either total or for
each item suggests that the items are sensitive and the measurement is appropriate as almost the full range of scores were
obtained. Additionally, caregivers did not consistently miss any particular item nor was there skewing of scores towards any
particular item or items.

3.2. Test–retest reliability

Data on 52 test–retest sets were available. The mean time between each administration of the questionnaire was 8.65
days (SD: 3.08, range: 5–22 days). ICC was calculated for total subscale score and for each item. Table 2 provides a summary
of the descriptive statistics for each item and the ICC based on the test–retest data. The ICC was based on a one-way random
effects model where people effects were random as in most cases, each rater only rated one individual (Shrout & Fleiss,
1979).
The ICC for the CC subscale score is 0.8 with a range of 0.5–0.8 across individual items. The ICC for the QoL subscale score is
0.8 with a range of 0.7–0.8 across individual items. Caregiver’s concerns regarding the future of the person they care for was
rated highest and also achieved the highest test–retest reliability. In terms of QoL, the person’s communication skills within
the home achieved the highest test–retest reliability, followed by mobility. Chinn (1991) recommends that measures should
have an ICC of at least 0.6 to be useful, as both subscale scores and the scores for most of the individual items surpass this
figure, it can be argued that the scale demonstrates good test–retest reliability. Indeed, only one item failed to reach this
threshold (lack of services, ICC = 0.5).

Table 1
Descriptive statistics for scale and item scores on the CC-QoLS at T1 (N = 99).

Mean (SD) Range

Caregiver’s concerns (CC) Total CC subscale score 11.71 (7.12) 0–29

Physical health 1.16 (1.17) 0–4
Mental health 1.27 (1.25) 0–4
Self-esteem 1.07 (1.16) 0–4
Aggression 1.89 (1.20) 0–4
Missing work/school/day care 0.69 (1.14) 0–4
Risk of accident 1.45 (1.24) 0–4
Future 2.17 (1.44) 0–4
Lack of services 1.99 (1.56) 0–4

Quality of life (QoL) Total QoL subscale score 19.67 (5.85) 6–31
Relationships with others within the home 2.94 (1.00) 1–4
Relationships with others outside the home 2.47 (1.21) 0–4
Leisure and recreation opportunities 2.26 (1.15) 0–4
Communication skills within the home 2.79 (1.01) 0–4
Communication skills outside the home 2.06 (1.24) 0–4
Self care skills 1.92 (1.18) 0–4
Mobility 2.74 (1.01) 0–4
Access to community facilities 2.48 (1.17) 0–4
2334 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

Table 2
Summary of the scale and items scores for each rating at T1 (N = 52).

Rating 1 Rating 2 ICC (single 95% CI

measures)
Mean (SD) Mean (SD)

Caregiver’s Concerns (CC) Total CC subscale score 12.65 (6.96) 11.69 (6.47) 0.81 0.69–0.89
Physical health 1.06 (0.94) 1.02 (0.98) 0.77 0.63–0.86
Mental health 1.27 (1.24) 1.37 (1.05) 0.70 0.53–0.82
Self esteem 1.29 (1.19) 1.21 (1.13) 0.71 0.55–0.83
Aggression 1.85 (1.26) 1.67 (1.17) 0.62 0.42–0.76
Missing work/school/day care 0.98 (1.31) 0.71 (1.05) 0.61 0.41–0.75
Risk of accident 1.67 (1.29) 1.50 (1.23) 0.73 0.57–0.84
Future 2.35 (1.30) 2.25 (1.28) 0.83 0.73–0.90
Lack of services 2.19 (1.37) 2.19 (1.95) 0.46 0.21–0.65

Quality of life (QoL) Total QoL subscale score 19.42 (6.19) 18.21 (6.14) 0.80 0.70–0.88
Relationships with others within the home 2.94 (1.04) 2.67 (0.90) 0.66 0.47–0.79
Relationships with others outside the home 2.44 (1.13) 2.38 (1.07) 0.77 0.63–0.86
Leisure and recreation opportunities 2.13 (1.07) 2.15 (1.00) 0.67 0.48–0.79
Communication skills within the home 2.79 (1.14) 2.62 (1.03) 0.65 0.47–0.78
Communication skills outside the home 1.98 (1.29) 2.00 (1.21) 0.81 0.69–0.89
Self-care skills 1.92 (1.22) 1.88 (1.17) 0.76 0.61–0.85
Mobility 2.73 (0.99) 2.54 (1.08) 0.79 0.66–0.87
Access to community facilities 2.48 (1.18) 2.12 (1.15) 0.67 0.49–0.80

3.3. Inter-rater reliability

Data were available for 50 sets of two raters. The relationships of the second caregiver included another family member or
a paid care worker and they either worked or resided in the same household as the participant in the large majority of cases.
Table 3 provides a summary of the descriptive statistics for each item and the ICC based on the inter-rater data. The ICC was
based on a one-way random effects model where people effects were random as in most cases, each rater only rated one
individual.
The ICC for the CC subscale scores is 0.67 with a range of 0.31 to 0.63 across the individual items. The ICC for the QoL
subscale scores is 0.63 with a range of 0.31 to 0.65 across the individual items. The items relating to caregiver’s concerns over
physical health and QoL in terms of access to community facilities showed the lowest inter-rater reliability. Caregiver’s
concerns regarding the aggression of the person they care for and her/his future was scored most highly and these items also
showed the highest inter-rater reliability. The items relating to QoL in terms of relationships with others outside the home
and self-care skills achieved the highest inter-rater reliability in this subscale. As both subscale scores surpassed an ICC of 0.6,
it can be argued that both are reliable measures with consistency between raters.

Table 3
Summary of the scale and items scores for each rater at T1 (N = 50).

Rater 1 Rater 2 ICC (single 95% CI

measures)
Mean (SD) Mean (SD)

Caregiver’s Concerns (CC) Total CC subscale score 12.02 (6.46) 12.88 (5.91) 0.67 0.50–0.80
Physical health 1.26 (0.97) 1.26 (0.97) 0.31 0.04–0.54
Mental health 1.26 (1.21) 1.68 (1.06) 0.51 0.28–0.70
Self esteem 1.20 (1.13) 1.36 (1.31) 0.53 0.30–0.70
Aggression 1.96 (1.28) 2.16 (1.18) 0.59 0.38–0.75
Missing work/school/day care 0.82 (1.21) 0.78 (1.04) 0.51 0.27–0.69
Risk of accident 1.52 (1.25) 1.58 (1.07) 0.49 0.25–0.67
Future 2.14 (1.28) 2.08 (1.21) 0.63 0.43–0.77
Lack of services 2.08 (1.40) 1.96 (1.20) 0.60 0.39–0.75

Quality of life (QoL) Total QoL subscale score 19.58 (5.78) 18.30 (5.08) 0.63 0.43–0.77
Relationships with others within the home 2.86 (1.03) 2.68 (0.98) 0.48 0.23–0.66
Relationships with others outside the home 2.36 (1.17) 2.24 (1.02) 0.65 0.46–0.79
Leisure and recreation opportunities 2.24 (1.12) 2.18 (1.04) 0.42 0.17–0.63
Communication skills within the home 2.78 (1.04) 2.60 (0.95) 0.48 0.24–0.67
Communication skills outside the home 2.06 (1.24) 2.20 (1.07) 0.60 0.39–0.75
Self-care skills 1.88 (1.15) 1.64 (1.01) 0.65 0.45–0.78
Mobility 2.84 (0.91) 2.50 (1.17) 0.51 0.27–0.69
Access to community facilities 2.56 (1.15) 2.28 (1.07) 0.31 0.04–0.54
 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340 2335

Table 4
Correlation coefficients (rs) for concurrent validity.

Weighted MOAS ABC-I Caregiver’s uplift Caregiver’s burden

** ** *
Caregiver’s Concerns 0.41 0.46 0.30 0.57*
Quality of life 0.22* 0.21* 0.30 0.26

* p  0.05.
** p  0.01.

3.4. Internal consistency

Cronbach’s alpha showed good correlation coefficients which were 0.85 for CC subscale and 0.80 for QoL subscale. An
inspection of the Cronbach’s alpha when each item was not included in the calculation indicates that all the items
contributed towards this reliability. All items apart from one, namely the mobility item on the QoL subscale, increased the
alpha coefficient. Removal of the mobility item on the QoL subscale increased alpha to 0.81. However, this change is very
small and therefore it does not warrant the mobility item to be deleted from the scale (Field, 2005).
Analysis of the inter-item correlations for each subscale indicates that all items inter-relate reasonably well (rs range:
0.22–0.64 for CC subscale; 0.04 to 0.71 for QoL subscale) apart from the item mobility with relationships with others within
the home and outside the home (0.09 and 0.04 respectively), however, this did not justify deletion of these items, as
discussed previously. A common cut-off for adequate item-total correlation is a coefficient of 0.3 (Field, 2005). All the items
in the CC subscale correlated adequately with total score (rs range: 0.53–0.66). The mobility item in the QoL subscale
achieved a coefficient of 0.28, the other items were all adequately correlated (rs range: 0.39–0.73). As the item-total
correlation is only just below the recommended cut-off, we felt it is acceptable to retain this item in the scale.

3.5. Split half reliability

The Spearman–Brown correlation coefficients for CC and QoL subscales were 0.81 and 0.70 respectively.

3.6. Correlation between CC and QoL

Bivariate correlations of the total subscale scores indicate that whilst the two subscale scores are related, they remain
distinct. There was a significant correlation between the two subscale total scores (p < 0.01), however the correlation
coefficient of 0.39 indicates a moderate relationship.

3.7. Construct validity – concurrent validity

Table 4 presents the results of the correlation analyses to investigate the relationships between CC, QoL and severity of
aggression and family caregiver’s outcomes. CC was significantly related to severity of aggression, caregiver’s uplift and
burden (p < 0.05). QoL was significantly related to both measures of severity of aggression (p < 0.05) but was not related to
either caregiver’s concerns or uplift. The correlation coefficients indicate small to medium effect sizes.

3.8. Change of score over time

CC-QoLS data on 61 participants were available at T1, T2 and T3. QoL scores remained stable over time (T1 mean = 19.34
(SD 5.79); T2 mean = 20.36 (SD 5.38); T3 mean = 20.51 (SD 4.63); F(2, 120) = 2.67, p = .07, n = 61). CC scores significantly
reduced over time (x2(2) = 13.12, p = .001, n = 61).

4. Discussion

4.1. General discussion

The UK Medical Research Council (www.mrc.ac.uk) stipulates that a new questionnaire should be simple to use, not take
long to complete, re-usable in different settings, capable of being completed by a proxy respondent, sensitive and specific to
the condition for which used, valid, and reliable. The CC-QoLS fulfils all these criteria and has demonstrated good
psychometric properties. The face validity of the CC-QoLS was guaranteed as the items for the CC-QoLS were gathered from
patient and caregiver consultation, and real life concern in a clinic based population. This bottom up approach to developing
rating scales is becoming a gold standard now in questionnaire development.
Several authors have suggested a number of core domains of QoL for people with ID (Schalock et al., 2002; Verdugo et al.,
2005). Townsend-White et al. (2012) report eight core domains of QoL that have been identified and extensively validated,
namely, emotional well-being, interpersonal relationships, material well-being, personal development, physical well-being,
self-determination, social inclusion and rights. The items on the CC-QoLS were selected to broadly cover these domains
2336 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

whilst keeping the questionnaire short, quick to administer and unambiguous. Indeed, Townsend-White et al. (2012)
suggest that instrument brevity is key feature. Some of these domains were operationalised to provide more objective
questions that are relevant to people with ID. For example, access to community facilities broadly covers the concepts of self-
determination, social inclusion and rights.
We attempted to cover all the core QoL domains mentioned in the literature in relation to generic or whole person QoL
in ID (Brown & Brown, 2003), however, it is important to note that the CC-QoLS is the first ever health-related QoL
(HRQoL) measure (see discussion in Bertelli et al., 2013) for adults with ID which is different from generic or whole person
QoL (Brown & Brown, 2003). Therefore, the items are specifically geared to address issues related to mental health,
specifically problem behaviour, such as aggression-related QoL and caregiver’s concerns. However, although specific to
health, the CC-QoLS does not include simply health symptom-based items as used in many scales in the health field
(Bertelli & Brown, 2006). The term HRQoL originated to define the perception of health specifically correlated to well-
being and functioning (EuroQoL Group, 1990). HRQoL has become an integral part of outcome measure in relation to
medical interventions such as pharmacological treatments of physical and mental health conditions. Bertelli et al. (2013)
in their systematic review did not detect any HRQoL used in any pharmacological intervention studies of people with
Autism Spectrum Disorder or ID. Therefore, the CC-QoLS may be a useful addition to outcome measures in such studies to
assess well-being and functioning.
Whilst it is acknowledged that there will be some overlap between the subscales as caregiver’s level of concern may
affect how they perceive the QoL of the person they care for, it is argued that both CC and QoLS in the CC-QoLS are
distinct constructs and important to measure. It has been established that aggressive behaviours have a significant
impact on caregivers (Unwin & Deb, 2011), therefore, it is important to consider the effects on caregivers as well as the
person themselves when measuring outcome, including when evaluating interventions. An effective intervention may
be one which does not have any overt impact on the behaviour but which supports caregivers, making them feel more
able to cope, and more confident in supporting the person. This may then have a positive impact on both the caregiver’s
and the person’s QoL as the caregiver may be more inclined to support the person in accessing more day activities or the
placement security for the person may be improved by reducing the risk of placement breakdown. A negative effect of
behaviour on the caregiver, on the other hand, may perpetuate problem behaviours exhibited by the person with ID.
Furthermore, existing research recommends that outcome measures for people with ID should include both the
perspective of the person and family members and/or service providers (Townsend-White et al., 2012). Therefore, we
think that even the CC subscale on its own is an important outcome measure following an intervention for aggressive
behaviour in people with ID.
Most QoL measures used in ID are self-rated (Townsend-White et al., 2012) and contain a large number of items, which
are not possible to use for people with severe and profound ID who have limited communication skills. Therefore, Verdugo
et al. (2014) have recently developed an informant-rated scale for use among people with severe and profound ID. Although
self-rated QoL scales could be used for adults with less severe ID, it is difficult to develop a QoL measure which could be used
across adults with all levels of severity of ID. Therefore, different measures of QoL may have to be used for different groups
which will lead to loss of consistency in studies. Therefore, we decided to use an informant-based measure for QoL which is
complemented by a caregiver’s concerns measure within the same scale. This will allow consistency of use of the same scale
(CC-QoLS) thus eliminating the need for different scales to be used for people with different levels of ID.
A week’s time-lapse between assessments was chosen for test re-test reliability assessment to minimise contamination
so that the caregiver could not recall their scores from initial completion whilst also minimising the chance of a significant
event which would genuinely affect caregiver’s concerns or the QoL of the person they care for, such as major illness or
implementation of a successful intervention. Some recommend a longer time lapse, for example, three months (Kline, 1992).
However, longer time scales are more appropriate for relatively stable, unchanging constructs such as intelligence quotient.
Caregiver’s concerns and QoL are likely to be rather transient and open to change and therefore a shorter time frame was
deemed more appropriate; indeed, the CC-QoLS was in fact designed to identify change over time.
Inter-rater reliability is particularly important in relation to the present questionnaire as caregivers are asked to rate the
QoL of the person they care for. QoL is often deemed a highly subjective construct therefore perceived QoL could vary from
rater to rater. For the questionnaire to be reliable, such rater effects would need to be minimal. The CC-QoLS performed well
in relation to inter-rater reliability, however, as would be expected, inter-rater reliability was better for CC. However, it was
still at acceptable levels for QoL.
Criterion-related validity was measured by assessing concurrent validity (whether similar results achieved on the
questionnaire correlated with results on other scales administered at T1). Ideally, concurrent validity would be assessed
against a ‘gold standard’ in the field (Streiner & Norman, 2003). However, in the absence of such a gold standard, correlations
with the other scales administered in the present study (weighted MOAS score, ABC-I score, family caregiver’s uplift score
and family caregiver’s burden score) were used instead. Whilst the other scales assessed different constructs such as
aggressive behaviour and caregiver’s outcomes, it can be argued that the constructs of caregiver’s concerns and participant’s
QoL may be related to these, with increasing concerns being associated with increasing severity of aggression. Indeed,
studies have demonstrated significant positive, linear relationships between severity of aggression and family caregiver’s
burden or paid carer’s burnout (Mills & Rose, 2011; Unwin & Deb, 2011; Vassos & Nankervis, 2012) and severity of aggression
and QoL of the individual (e.g. Murphy, 2009; Schalock et al., 2002). This approach is consistent with our aim of developing a
HRQoL rather than a generic QoL for adults with ID.
 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340 2337

Correlation coefficients indicate good internal consistency for the scale as an alpha coefficient of 0.7–0.8 is generally
accepted as demonstrating acceptable reliability (Field, 2005) especially considering each subscale only has eight items. As
Streiner and Norman (2003) suggest, Cronbach’s alpha is influenced by the number of items on a scale with longer scales
more likely to demonstrate homogeneity of items. Kline (1992) suggests that an alpha coefficient of 0.7 or lower can be
acceptable for some psychological tests due to the heterogeneity of some psychological constructs. Furthermore, Nunnally
(1978) suggests that alpha should probably not be above 0.90 as this could indicate item redundancy.
None of the inter-item correlations within each subscale had a correlation coefficient over 0.8–0.9. Higher coefficients
could indicate problems with multicollinearity or singularity (Field, 2005). Therefore, each item is related to each other item
on each subscale but each item remains distinct, measuring a different element of the construct.
The reliability coefficients indicate that the CC-QoLS is a reliable measure with good test–retest reliability, inter-rater
reliability and internal consistency. Test–retest reliability was generally higher than inter-rater which is perhaps
anticipated as there is likely to be less inconsistency over time than between people. Inter-rater reliability score could
be improved with appropriate training, which we were unable to provide because of lack of resources. Inspection of the
reliability of each individual item indicates that the majority of items demonstrate good reliability, apart from mobility
in the QoL subscale. Therefore, we recommend that the total subscale scores should be used in practice rather than
individual item scores.
The internal consistency analyses indicate that the items on each subscale are measuring the same underlying construct.
Furthermore, the correlation between the two subscales indicate that they are significantly negatively correlated, however,
the coefficient indicates a small to moderate effect which justifies the two factor structure of the questionnaire. The
subscales also relate to other measures of severity of aggression and outcomes for family caregivers as would be anticipated,
indicating concurrent validity.
We have carried out some preliminary analysis to test the sensitivity of CC-QoLS score change over time. Whereas the CC
subscale score showed significant change over time (from T1 to T3) the QoL subscale did not show such change. The
sensitivity to change should be assessed in future research within the context of a randomised controlled trial (RCT) design
where one intervention is compared with another or a placebo or treatment as usual. There is now an emphasis on the
assessment of cost effectiveness along with the clinical effectiveness of any intervention in healthcare (see
www.nice.org.uk). In the general non-ID population Quality Adjusted Life Years (QALYs) (Weinstein & Stason, 1977) is
used to measure cost-effectiveness of interventions. The common QoL instruments used for measuring QALY are EQ-5D
(EuroQoL Group, 1990) and SF-6D (Brazier et al., 2002). However, neither EQ-5D nor SF-36 have been validated for use
among adults with ID. Therefore, there was an urgent need to develop a HRQoL for adults with ID which could be used for
both clinical as well as cost-effectiveness analysis, which could now be fulfilled with the CC-QoLS. However, the CC-QoLS’s
efficacy to measure cost effectiveness and QALY has to be assessed in future research.

4.2. Limitations

The procedure for development of items for the CC-QoLS is not as stringent as we would have desired. Ideally we wanted
to use a more robust method of data collection at the outset by interviewing people with ID who show aggression and their
caregivers, and then use qualitative methodology to gather items for CC-QoLS. We have used a similar methodology while
developing Dementia Screening Questionnaire for Individuals with ID (DSQIID) (Deb, Hare, Prior, & Bhaumik, 2007) when we
interviewed caregivers of people with Down syndrome who developed dementia and analysed data qualitatively in order to
develop items for the scale. We have used a similar methodology to develop Patient-Head Injury Participation Scale (P-HIPS)
(Deb et al., 2007a) and the caregiver version Carer-Head Injury Participation Scale (C-HIPS) (Deb et al., 2007b) in which we
carried out in depth interviews with patients with head injury and their caregivers in order to analyse interview data using
qualitative methodology to develop the items in the questionnaires. Similarly, Verdugo et al. (2014) used a very extensive
consultation method with experts and professionals in the field including Delphi method and qualitative analysis in
developing the San Martı́n QoL scale for people with ID. It would have been ideal for us to use a similar methodology for item
development for the CC-QoLS.
It would have been ideal to develop a measure which could be completed both by the person with ID and also a caregiver.
However, the items in CC-QoLS are chosen carefully so that they could be rated easily by caregivers as there is no item that
relate directly to how a person with ID feels within her/himself, a question that would have been difficult to answer by a
caregiver.

Conflict of interest

None.

Acknowledgement

The study was funded by the Baily Thomas Foundation, UK. We are grateful to professionals who helped to recruit adults
with ID and their caregivers who took part in the study.
2338 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

Appendix A

Carergiver’s Concerns-Quality of Life Scale (CC-QoLS)

Patient identification: ……………………………….Name of caregiver………………………

Date: …………………………………………………Name of rater…………………………..

Carergiver’s Concerns (CC)

How concerned are you about the person you are caring for in relation to...?

Not Mildly Moderately Very Seriously

Concerned Concerned Concerned Concerned Concerned
(0) (1) (2) (3) (4)
Her/his physical health

Her/his mental health

Her/his self esteem

Her/his aggression

Missing work/school/day care

Risk of accident

Her/his future

Lack of services

Aggregated column score

Total CC score

Quality of Life (QoL)

How would you qualify the person you are caring for in relation to...?

Very poor Poor Satisfactory Good Very Good

(0) (1) (2) (3) (4)

Relationships with others

(within home)

Relationships with others

(outside home)
 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340 2339

Leisure and recreation

opportunities

Communication skills
(within home)

Communication skills
(outside home)

Self care skills

Mobility

Access to community facilities

Aggregated column score

Total QoL score

References

Aman, M. G., Burrow, W. H., & Wolford, P. L. (1995). The Aberrant Behavior Checklist-Community: Factor validity and effect of subject variables for adults in group
homes. American Journal on Mental Retardation, 100(3), 283–292.
Bertelli, M., & Brown, I. (2006). Quality of life for people with intellectual disabilities. Current Opinion in Psychiatry, 19, 508–513.
Bertelli, M., Merli, M. P., Rossi, M., Lassi, S., Bianco, A., & Colangelo, J. (2013). Quality of life in pharmacological intervention on Autism Spectrum Disorders.
Advances in Mental Health and Intellectual Disabilities, 7(1), 40–48.
Brazier, J. E., Roberts, J. F., & Deverill, M. D. (2002). The estimation of a preference based measure of health from the SF-36. Journal of Health Economics, 21, 271–292.
Brown, I., & Brown, R. I. (2003). Quality of life and disability: An approach for community practitioners. London: Jessica Kingsley.
Chang, L. (1994). A psychometric evaluation of 4-point and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18,
205–216.
Chinn, S. (1991). Repeatability and method comparison. Thorax, 46, 454–456.
Clark-Carter, D. (2001). Doing quantitative psychological research: From design to report. Hove: Psychology Press Ltd.
Crocker, A. G., Mercier, C., Lachapelle, Y., Brunet, A., Morin, D., & Roy, M. E. (2006). Prevalence and types of aggressive behaviour among adults with intellectual
disabilities. Journal of Intellectual Disability Research, 50(9), 652–661.
Cummins, R. A. (1997). Self-rated quality of life scales for people with an intellectual disability: A review. Journal of Applied Research in Intellectual Disabilities,
10(3), 199–216.
Deb, S. (2013). Psychopharmacology. In ‘Handbook of Psychopathology in Intellectual Disability: Research, Practice, and Policy, Autism and Child Psychopathol-
ogy Series’ (Eds. E. Tsakanikos & J. McCarthy) Springer Science + Business Media, New York, USA (book chapter, 19). (Series Editor: Johnny L. Matson). Pp.
307–324.
Deb, S., Bryant, E., Morris, P. G., Prior, L., Lewis, G., & Haque, S. (2007a). Development and psychometric properties of the Patient-Head Injury Participation Scale (P-
HIPS) and the Patient-Head Injury Neurobehavioral Assessment Scale (P-HINAS): Patient and family determined outcome scales. Neuropsychiatric Disease and
Treatment, 3(3), 373–388.
Deb, S., Bryant, E., Morris, P. G., Prior, L., Lewis, G., & Haque, S. (2007b). Development and psychometric properties of the Carer-Head Injury Neurobehavioral
Assessment Scale (C-HINAS) and the Carer-Head Injury Participation Scale (C-HIPS): Patient and family determined scales. Neuropsychiatric Disease and
Treatment, 3(3), 389–408.
Deb, S., Hare, M., Prior, L., & Bhaumik, S. (2007). Dementia Screening Questionnaire for Individuals with Intellectual Disabilities (DSQIID). British Journal of
Psychiatry, 190, 440–444.
EuroQoL Group (1990). EuroQoL – A new facility for the measurement of health related quality of life. Health Policy, 163, 199–208.
Field, A. (2005). Discovering statistics using SPSS for Windows. London: Sage Publications.
Hemmings, C., Deb, S., Chaplin, E., Hardy, S., & Mukherjee, R. (2013). Research for people with intellectual disabilities and mental health problems: A view from the
UK. Journal of Mental Health Research in Intellectual Disability, 6(2), 127–158.
2340 G. Unwin, S. Deb / Research in Developmental Disabilities 35 (2014) 2329–2340

Kline, P. (1992). The handbook of psychological testing. London: Routledge.

Lissitz, R. W., & Green, S. B. (1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10–13.
Mills, S., & Rose, J. (2011). The relationship between challenging behaviour, burnout and cognitive variables in staff working with people who have intellectual
disabilities. Journal of Intellectual Disability Research, 55(9), 844–857.
Murphy, G. (2009). Challenging behaviour: A barrier to inclusion? Journal of Policy and Practice in Intellectual Disabilities, 6(2), 89–90.
Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.
Oliver, P. C., Crawford, M. J., Rao, B., Reece, B., & Tyrer, P. (2007). Modified Overt Aggression Scale (MOAS) for people with intellectual disability and aggressive
challenging behaviour: A reliability study. Journal of Applied Research in Intellectual Disabilities, 20(4), 368–372.
Pruchno, R. (1990). The effects of help patterns on the mental health of spouse caregivers. Research on Ageing, 12, 57–71.
Ratey, J. J., & Gutheil, C. M. (1991). The measurement of aggressive behaviour: Reflections on the use of the Overt Aggression Scale and the Modified Overt
Aggression Scale. Journal of Neuropsychiatry, 3, 557–560.
Schalock, R. L., Brown, I., Brown, R., Cummins, R. A., Felce, D., Matikka, L., et al. (2002). Conceptualization, measurement, and application of quality of life for
persons with intellectual disabilities: Report of an international panel of experts. Mental Retardation, 40(6), 457–470.
Shedlack, K. J., Hennen, J., Magee, C., & Cheron, D. M. (2005). A comparison of the Aberrant Behavior Checklist and the GAF among adults with mental retardation.
Psychiatric Services, 56(4), 484–486.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
Silver, J. M., & Yudofsky, S. C. (1991). The Overt Aggression Scale: Overview and guiding principles. Journal of Neuropsychiatry, 3, 522–529.
Streiner, D. L., & Norman, G. R. (2003). Health measurement scales: A practical guide to their development and use. Oxford: Oxford University Press.
Townsend-White, C., Pham, A. N. T., & Vassos, M. V. (2012). A systematic review of quality of life measures for people with intellectual disabilities and challenging
behaviour. Journal of Intellectual Disability Research, 56(3), 270–284.
Tyrer, P., Oliver-Africano, P. C., Ahmed, Z., Bouras, N., Cooray, S., Deb, S., et al. (2008). Risperidone, haloperidol, and placebo in the treatment of aggressive
challenging behaviour in patients with intellectual disability: A randomised controlled trial. Lancet, 371, 57–63.
Tyrer, P., Oliver-Africano, P. C., Ahmed, Z., Bouras, N., Dickens, S., Cooray, S., et al. (2009). Neuroleptics in the treatment of aggressive challenging behaviour for
people with intellectual disabilities (NACHBID trial). Health Technology Assessment (HTA) Monograph. National Institute of Health Research (NIHR), April 2009,
13(21), 1–92 Department of Health, UK.
Unwin, G., & Deb, S. (2008). Psychiatric and behavioural assessment scales for adults with learning disabilities. Advances in Mental Health in Learning Disability,
2(4), 37–45.
Unwin, G., & Deb, S. (2009). The effectiveness of interventions on aggression in adults with intellectual disabilities: A naturalistic observational study. World
Psychiatry, 8(S1), 237.
Unwin, G., & Deb, S. (2011). Family caregiver uplift and burden: Associations with aggressive behaviour in adults with intellectual disability. Journal of Mental
Health Research in Intellectual Disability, 4(3), 186–205.
Vassos, M. V., & Nankervis, K. L. (2012). Investigating the importance of various individual, interpersonal, organisational and demographic variables when
predicting job burnout in disability support workers. Research in Developmental Disabilities, 33(6), 1780–1791.
Verdugo, M. A., Gómez, L. E., Arias, B., Navas, P., & Schalock, R. L. (2014). Measuring quality of life in people with intellectual and multiple disabilities: Validation of
the San Martı́n scale. Research in Developmental Disabilities, 35, 75–86.
Verdugo, M. A., Schalock, R. L., Keith, K. D., & Stancliffe, R. J. (2005). Quality of life and its measurement: Important principles and guidelines. Intellectual Disability
Research, 49(10), 707–717.
Weinstein, M., & Stason, W. (1977). Foundations of cost-effectiveness analysis for health and medical practices. New England Journal of Medicine, 296(13), 716–721.
World Health Organization (1980). ICIDH: International classification of impairment, disabilities and handicap. A manual of classification relating to the consequence of
disease. Geneva: WHO.
World Health Organization (1997). ICIDH-2: International classification of impairment, activities, and participation. A manual of disablement and functioning. Geneva:
WHO.
World Health Organization (2012). Measuring health and disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0). Geneva: WHO.