See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/44681052

A Caregiver–Child Socioemotional and Relationship Rating Scale (CCSERRS)

Article  in  Infant Mental Health Journal · March 2010

DOI: 10.1002/imhj.20252 · Source: PubMed

CITATIONS READS
21 462

3 authors, including:

Christina J. Groark
University of Pittsburgh
41 PUBLICATIONS   589 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Institutionalization View project

The Fairstart Foundation training programmes at www.fairstartfoundation.com View project

All content following this page was uploaded by Christina J. Groark on 11 January 2018.

The user has requested enhancement of the downloaded file.

 NIH Public Access
Author Manuscript
Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
Published in final edited form as:
NIH-PA Author Manuscript

Infant Ment Health J. 2010 March ; 31(2): 201–219. doi:10.1002/imhj.20252.

A Caregiver-Child Social/Emotional and Relationship Rating Scale

(CCSERRS)1

Robert B. McCall, Christina J. Groark, and Larry Fish

University of Pittsburgh Office of Child Development
Robert B. McCall: mccall2@pitt.edu; Christina J. Groark: cgroark@pitt.edu; Larry Fish: laf16@pitt.edu

Abstract
This paper reports the construction and pilot reliability, validity, and psychometric properties of a
new caregiver-child rating scale that emphasizes caregiver-child social-emotional interactions and
relationships. While the scale was developed and studied in the context of orphanages for young
children, it potentially could be used in non-residential early care and education settings as well as
NIH-PA Author Manuscript

for parent-child interactions in the home. The intent was to assess a few dimensions that
comprehensively cover the range of caregiver-child social-emotional interactions and relationships
but could be administered in a relatively short period of time in a variety of situations and would not
require extensive coder training, manuals, or materials. Results showed that the scale can be reliably
administered even using observation periods as short as five minutes, reliability was replicated over
seven different coders working in three different orphanages, and ratings of caregivers were similar
across different types of caregiving activities (i.e., feeding, dressing/bathing, free play) and for
caregivers attending to children birth to 4 and 4 to 8 yrs. of age. In the orphanage context, factor
analyses showed the scale primarily reflects caregiver-child mutual engagement and relationship
with subordinate components of caregiver punitiveness and caregiver- vs. child-directed behaviors
and intrusiveness.

Early social/emotional relationship experiences, especially warm, caring, sensitive, and

responsive interactions between adults and their infants and young children, are crucial
contributors to promoting attachment (e.g., DeWolff & van IJzendoorn, 1997; van IJzendoorn
& Sagi, 1999), which in turn is associated with longer-term positive child outcomes in social
and mental development (e.g., Aviezer, Sagi, Resnick, & Gini, 2002; Landry, Smith, & Swank,
NIH-PA Author Manuscript

2006; Landry, Smith, Miller-Loncar, & Swank, 1997; Landry, Smith, Swank, & Miller-Loncar,
2000; Steelman, Assel, Swank, Smith, & Landry, 2002). Social/emotional relationship
experiences have also been associated with the quality of early care and education
environments, early childhood developmental gains, and longer-term educational success (e.g.,
Edwards & Raikes, 2002; Kontos, Howes, Shinn, & Galinsky, 1995).

Conversely, insecure attachment, especially disorganized attachment, is related to increased

later problem behaviors, including externalizing behaviors in males and other social, behavior
control, crime, and mental health problems, more so in high-risk children and those who
continue to experience insensitive parenting and/or child care (e.g., Carlson, 1998; Fonagy et

1This project funded in part by NICHD grant HD39017 for the St. Petersburg, Russian Federation activities and by Whole Child
International (Karen Gordon, President) for the Nicaraguan activities. The authors are indebted to Auxiliadora Alvarado, Ian Downing,
Juan Diego Rivas, Raquel Suazo, and Karla Urbina for conducting the coding in Managua, to their supervisors Mauricio Gaborit and
Marisol Vanegas Guido of the Universities of Central America in San Salvador and Managua, respectively, and to Gabriella Serrano of
Whole Child International for logistics management.
 McCall et al. Page 2

al., 1997; Greenberg, 1999; Lyons-Ruth, Alpern, & Repacholi, 1993; Rothbaum & Weisz,
1994; Shaw, Owens, Vondra, Keenan, & Winslow, 1997).
NIH-PA Author Manuscript

More specifically, institutional rearing environments for young children tend to provide quite
minimal social/emotional relationship experiences (e.g., St. Petersburg-USA Orphanage
Research Team, 2005, 2008; Rosas & McCall, 2008), and perhaps as a partial consequence of
this deprivation such children tend to be substantially underdeveloped and have higher rates
of behavioral problems even after being adopted into advantaged families (Blizzard, 1990;
Gunnar, 2001; Johnson, 2000; MacLean, 2003; St. Petersburg-USA Orphanage Research
Team, 2005, 2008). Further, interventions that emphasize improved early social/emotional
relationship experiences in orphanages (St. Petersburg-USA Orphanage Research Team,
2008; Zeanah, Smyke, & Koga, 2003), provide high-quality foster care instead of
institutionalization (Nelson, Zeanah, Fox, Marshall, Smyke, & Guthrie, 2007), or promote
responsiveness in parents towards their own infants (Landry et al., 2006) have produced
improvements in children's development in several domains.

The Need for an Assessment Instrument for Social/Emotional Relationship Caregiver

Interactions
Despite the theoretical and empirical importance of early social/emotional relationships, there
are relatively few measures of children's home or early care and education environments that
focus specifically on those aspects of parent/caregiver-child interactions for children under 4-6
NIH-PA Author Manuscript

years of age.

Assessments of early care environments fall into roughly three categories (for a review, see
Melhuish, 2001), those that primarily measure the total environmental setting, those that focus
on individual children, and those that assess the nature of the caregiver-child interaction.

Measures of the total environmental setting—The most commonly used assessments

of early care and education environments are comprehensive and reflect the total environment
—the physical facility, toys and equipment, activities, staff support, and other aspects of the
care environment in addition to parent/caregiver-child interactions. This comprehensiveness
is a major asset, and these assessments typically do correlate with children's contemporary and
future development (Melhuish, 2001). But the subscales of these instruments that pertain
specifically to caregiver-child interactions are short and a small part of the total assessment.

For example, the HOME and ECERS are perhaps the most widely used general assessments.
The HOME Inventory (Bradley & Caldwell, 1995; Caldwell & Bradley, 1984), originally
designed to assess the home environment but adapted to group care (NICHD Study of Early
Child Care Manual), has several subscales including Responsivity, Acceptance, and
NIH-PA Author Manuscript

Involvement that reflect caregiver-child interactions. But these subscales involve caregiver
stimulation of children and the responsivity of caregivers to children's initiatives to a greater
extent than social-emotional relationships. Further, items are scored 0/1, so they are limited in
their ability to reflect extent and thus are relatively insensitive to gradations of social/emotional
interactions. Also, when the instrument is used in a group setting, the caregiver only needs to
display the indicated behavior once to one child to receive credit for that item.

Similarly, the environmental rating scales designed especially for assessing early care and
education environments (i.e., ITERS, ECERS, FDCERS; Harms, Clifford, & Cryer, 1998;
Harms, Cryer, & Clifford, 1990) primarily assess the global environment (subscales of Space
and Furnishings, Personal Care Routines, Activities, Program Structure, Staff Support) in
addition to caregiver-child social behavior (subscales of Listening, Talking/Language-
Reasoning, and Interaction). While scoring is on a 7-point (rather than 0/1) scale, only four
items pertain to language-reasoning in children and only five items to social interaction; no

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 3

items specifically measure social-emotional interaction or adult-child relationships. Moreover,

both these measures require about an hour or more of observation and scoring.
NIH-PA Author Manuscript

Assessments of individual children—Assessments that focus on individual children

(e.g., Melhuish, Mooney, Martin, & Lloyd, 1990; Pierrehumbert, Ramstein, Krucher, El-Najar,
Lamb, & Halfon, 1996; Sylva, 1997) have the obvious advantage of providing detailed
information about individual children, which the more general measures that aggregate over
all children would obscure. However, they require more time and energy to conduct, and do
not provide a total picture of adult-child interactions in group settings.

Assessments of caregiver-child interactions—Other scales focus more directly on

caregiver-child interactions for individual caregivers (e.g., Arnett, 1989; Erickson, Sroufe, &
Egeland, 1985; Goossens & Melhuish, 1996; Howes & Stewart, 1987; McGuire & Richman,
1987). Perhaps the most widely used is the Caregiver Interaction Scale (Arnett, 1989), which
consists of 26 items directly aimed at assessing caregiver-child interactions. A principal
component analysis of the items suggested that they reflected primarily authoritarian and
stimulating interaction (van IJzendoorn & Tevecchio, 1998); the scale places less emphasis on
the emotional component of interactions and relationships. Further, scoring is on a 4-point
scale, and given the current scoring directions, the assessment can reach a ceiling that limits
its ability to discriminate gradations in more positive behaviors (R. Fiene, personal
communication, October 22, 2007).
NIH-PA Author Manuscript

Purpose of this Project

This paper reports preliminary information on a new scale, called the Caregiver-Child Social/
Emotional and Relationship Rating Scale (CCSERRS) (pronounced “sirs”). The CCSERRS is
an attempt to assess fundamental aspects of caregiver-child social/emotional interactions and
relationships that is simple, contains only a few items (18), and requires a relatively short
amount of time to administer. While the current scale was primarily developed in orphanage
contexts, it can be used to rate individual parents at home as well as caregivers in non-residential
group settings.

This paper describes the development of the CCSERRS, presents some pilot reliability and
validity data, and gives reliability and psychometric data based on orphanage caregivers and
children. Although these psychometric data involve relatively small Ns, the assessment is not
fully standardized, and population “norms” are not provided (but neither are they for most other
environmental scales), we present this preliminary report to try to fill the assessment gap
described above and to stimulate other researchers to use the CCSERRS in different contexts,
assess its reliability and validity, and contribute to its overall psychometric foundation and
NIH-PA Author Manuscript

construct validity.

The Development of the CCSERRS

The Item Pool
Authors McCall and Groark made more than a dozen systematic observations of wards in three
orphanages for children birth to 4 years of age in St. Petersburg, Russian Federation, over a
six-year period beginning in 2001 as part of a large social/emotional relationship intervention
project (Groark, Muhamedrahimov, Palmov, Nikiforova, & McCall, 2005; Muhamedrahimov,
Palmov, Nikiforova, Groark, & McCall, 2004; St. Petersburg-USA Orphanage Research Team,
2005, 2008). The interventions produced substantial variation in caregiver-child social/
emotional interaction and relationships. The notes of these observations on caregivers and
children before and after the interventions were systematically reviewed, and all of the positive
and negative caregiver and children's behaviors were listed. In addition, items pertaining to

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 4

caregiver-child interactions from the HOME Inventory, the ITERS and ECERS Scales, the
Caregiver Interaction Scale, and behaviors rated by Koren-Karie, Sagi-Schwartz, and Egoz-
Mizrachi (2005) in their study of child care in Israel were added to the list. The items in this
NIH-PA Author Manuscript

“pool” were then subjectively sorted into groups that were relatively similar within groups but
dissimilar between groups, and then the general concept that characterized the underlying
construct in each group was identified.

These basic constructs were further refined according to several subjective criteria.

First, it was desirable to have as few constructs as possible that nevertheless covered the
maximum number of items in the pool to produce a comprehensive yet efficient set.

Second, the orphanage observations indicated that certain behaviors that appeared to be direct
opposites actually were not simply two ends of a continuum and could vary somewhat
independently of one another. Thus, separate constructs for each were included.

Third, caregivers sometimes deliberately behaved in ways they thought observers were looking
for, or at least they displayed their “best” caregiving behaviors, but it was clear from the
children's behavior that these “model” caregiver behaviors were unusual. Therefore, it seemed
necessary to include some children's behaviors in a scale that represented caregiver-child
interactions and relationships.
NIH-PA Author Manuscript

The CCSERRS
Appendix I presents the 18 items on the Caregiver-Child Social/Emotional and Relationship
Rating Scale clustered into four caregiver categories of caregiver engagement (negative,
positive), caregiver/child-directed behaviors, behavioral control (negative, positive), and
caregiver affect plus three child categories of child engagement, child affect, and child
relationship with the caregiver. Table 1 presents the general construct for each item followed
by behaviors illustrating the concept that might be observed for caregivers attending to children
a few months to approximately 6 years of age. The scale is also available in Spanish from the
authors.

Ratings
Each item is scored on a four-point scale (0, 1, 2, 3) reflecting the frequency with which that
behavior (positive or negative) occurred relative to the potential opportunities for it to
occur. Items were rated 0=never, 1=rarely, 2=frequently, and 3=consistently in terms of their
frequency during an observation relative to the opportunity for the behavior to occur. For
example, a caregiver might not display behavioral control or punish a child unless a child
happens to deviate from expected behavior. If no deviant behavior provoked punishment, the
NIH-PA Author Manuscript

item would be scored 0, because the caregiver did not control or punish children. However, if
only one child deviated from expected behavior and the caregiver punished that child, that
would be scored 3 because it happened on each opportunity.

An exception to this rating strategy was made for items assessing caregivers' or children's
responses to the other. If children played by themselves during the entire observation and never
bid for a caregiver's attention, there was no opportunity for the caregiver to respond, but such
an item was also scored 3 because the caregiver consistently “failed to respond” (i.e., there was
no caregiver response to children). The same rating strategy was used for the item “child
responsiveness and anticipation.”

Notice that items 1, 2, 5, 7-9, 11, 14, and 16 are all negative or undesirable behaviors while
the remainder of the items are positive. Scoring, however, reflects the relative extent to which
the behavior named in the item was observed, regardless of whether it was positive or

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 5

negative. Consequently, negative behaviors must be reverse scored after the initial ratings have
been completed but before scores are added to create a total or subscale scores. Of course, users
of the CCSERRS may elect to score all items in a positive direction, but this leads to the
NIH-PA Author Manuscript

awkward double-negative definition of negative items, for example, “the lack of caregiver
detachment” or the “lack of failure to respond.”

Observation period
No “standardized” observation period and circumstances have been prescribed. Instead, the
instrument is quite general, and could be used to rate caregiver behavior in a free play context
for as little as five minutes for example, or caregiver behavior in feeding, bathing/dressing,
and free play situations each for 5 minutes or longer. The lack of a standardized observation
period has the liability of possibly limiting direct comparisons of scores across different
contexts and observational procedures. However, such comparisons were made in this paper,
and it is not clear how often researchers would actually make direct comparisons if the
procedures were standardized. Further, the lack of standardized observation criteria means that
the scale can be used in a variety of contexts and for different purposes specific to an
intervention or site.

Scoring
After appropriate items have been reverse scored, the 18 scores can simply be added and then
NIH-PA Author Manuscript

averaged, giving an overall index ranging between 0 and 3.

Pilot Reliability
Method
Authors McCall and Groark conducted pilot reliability assessments on 12 arbitrarily selected
caregivers for children birth to 4 yrs., 4 in a no-intervention orphanage and 8 in an orphanage
that had an intervention consisting of training plus structural changes aimed at improving
caregiver-child social/emotional interactions (for a complete description, see St. Petersburg-
USA Orphanage Research Team, 2008). Although small and unbalanced across intervention
conditions, this pilot sample provided a wide range of caregiver-child interaction behaviors on
most items on which to determine if the scale had the potential to be scored reliably.

Results
Percent agreement—On the 216 paired ratings of 18 items over 12 caregivers, the raters
assigned identical ratings on 61% of the cases and were identical or within 1 point on 96% of
the cases.
NIH-PA Author Manuscript

Total scores—For the Total Scores, the average difference between the two raters was 1.44,
which was 4% of the average Total Score of 34.56 (the difference per item between raters
averaged only .08 on a scale ranging from 0-3.00). The correlation between raters across the
12 caregivers was .94.

Item reliabilities—The correlations between raters and their mean scores across caregivers
were calculated for each of the 18 items.2 Mean differences between raters and the correlation
between raters provide two different kinds of indices of inter-rater consistency. Correlations
were low (rs in the .50s, p < .10) for only three items, and no item showed both a mean difference
and a low correlation between raters. Thus, even at the individual item level and on this small
pilot sample, the potential to score these items consistently across raters seemed acceptable.

2A more complete report including tables of data is available at (www.educaton.pitt.edu/ocd).

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 6

Pilot Validity
Method
NIH-PA Author Manuscript

Authors McCall and Groark rated 17 arbitrarily selected caregivers in the no-intervention and
36 caregivers in the training plus structural change intervention orphanages. The intervention
emphasized warm, caring, sensitive, and responsive caregiver-child interactions, and was
successful at improving HOME Inventory scores and improving children's physical, mental,
and social-emotional development (St. Petersburg-USA Orphanage Research Team, 2008).
Mean differences in ratings between these two orphanages would contribute to the proposition
that the CCSERRS is sensitive to quasi-experimentally produced differences in caregiver
social-emotional actions and relationship behavior (i.e., validity).

Results
Table 1 presents means and standard deviations for individual items for the non-intervention
and intervention orphanages for each of the 18 items, the independent-samples t test of mean
differences and its significance, and the partial eta2 reflecting the percent variance associated
with the mean difference between the two orphanages.

Table 1 shows significant orphanage differences for all but four individual items, which had
no or very limited variability. Most of the partial eta2 estimates of percent variance associated
with the mean difference between orphanages were above .50 (except the items that had limited
NIH-PA Author Manuscript

variability), meaning that half the total variability in individual item ratings was associated
with orphanage differences. The Total Score was more than twice as great in the intervention
group—a mean item score of 2.39 in vs. 1.06.

These results suggest that the individual items and the Total Score on the CCSERRS are
sensitive to quasi-experimentally produced improvements in caregiver behavior.

Formal Reliability in a Nicaraguan Orphanage

Reliability was more formally assessed by several independent coders in an orphanage for
children a few months to 8 yrs. old in Managua, Nicaragua. The orphanage environment was
spartan with minimum furnishings, toys, and equipment. For example, the average Infant-
Toddler Environmental Rating Scale and Early Childhood Environmental Rating Scale scores
were 1.16 on scales ranging from 1-7.

Method
For purposes of reliability, there were five coders who formed ten pairs who coded 14 different
NIH-PA Author Manuscript

caregivers who were responsible for providing all aspects of care for children.

Coder training—Coders were trained by first spending three days observing caregiving
activities that had been videotaped in a different orphanage and in USA child care settings and
discussing the nature of behaviors on the tapes with respect to the 18 items on the CCSERRS.
This was followed by one morning of live practice observation in an elementary school center;
each coder possessed definitions of the 18 items, pairs of coders observed a caregiver but
assigned ratings privately, and then pairs of coders discussed any discrepancies between their
ratings.

Observations—Ideally, each caregiver was observed engaged in three types of Activities—

feeding, dressing/bathing, and free play. Within each Activity, the caregiver was observed for
two 5-min. observations or Episodes while engaged in that Activity with two different children
in each Activity. Thus, each caregiver was observed in six 5-min. Episodes of three different

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 7

Activities for a total of 30 min. Mean scores were calculated for each Activity (the score from
one Episode was used if it was impossible to obtain two Episodes on a caregiver; 27% of
Episodes were missing as well as for Total Score.
NIH-PA Author Manuscript

Reliability Results
Percent agreement—Over all items and all Episodes, pairs of coders assigned an identical
rating in 56.44% of the cases and were identical or within one point on 90.57% of the cases,
figures only slightly lower than the 61% and 96% for the coders in the pilot study. Agreement
was slightly higher for the Feeding Activity than for either the Bathing/Dressing or Free Play
Activities. Percent agreements did not differ as a function of whether the caregiver was in
charge of children birth to 4 yrs. or 4-8 yrs. of age.

For individual items, between 43% and 71% of the rating pairs were identical and 81%-98%
were identical or within one point. Each coder had percent-agreement rates with each of the
other coders similar to those reported above for the entire group, indicating that none of the
five coders seemed to be at odds with any of the others.

Psychometric Information
The five coders individually rated all 55 of the major caregivers in the orphanage who attended
to children 6 yrs. of age or younger. It was of interest to know how much variability in the
NIH-PA Author Manuscript

scores could be attributed to differences between caregivers, coders, and the three Activities.
An analysis of variance with Caregivers and Coders as random and Activity as a fixed factor
revealed a significant effect for Caregivers (F=3.07, df=54/267, p<.001) and Activity
(F=11.69, df=2/267, p<.001), but not for Coders (F<1). Differences between Caregivers
accounted for 38% of the variance (partial eta2), Activity was associated with 8%, and Coders
with 1%. This indicated quite clearly that the ratings reflected individual differences in
Caregivers to a substantially greater extent than differences between Coders (38% versus 1%;
Groark, McCall, & the Whole Child International Team, 2007).

Item—Partial Total Score correlations—The correlations between each item and the
Total Score based on the 17 other items excluding the item in question were calculated. Except
for “5. Caregiver-directed behaviors” (.315), “16. Child negative affect” (.426), “15. Child
responsiveness and anticipation” (.457), and “11. Caregiver negative affect” (.476), all
correlations were above .50 and eight items correlated above .60. Items with lower correlations
are primarily negative behaviors, have relatively lower reliabilities, and occurrences of these
behaviors were either so infrequent (e.g., children and caregivers rarely displayed negative
affect) or very frequent (i.e., caregiver-directed behavior) that variability was substantially
constrained. Generally, the Total Score seems to reflect a single general characteristic of
NIH-PA Author Manuscript

caregiver behavior, but separate less prominent characteristics may also be involved (see
below).

Factor analysis—The ratings of 55 caregivers by five coders were subjected a a principal

component analysis with varimax rotation, and Table 2 presents all factor loadings .30 or
greater. Negative behavioral items were reverse coded so high scores and positive loadings
reflect the positive version of the behavior in question.

The first of three factors retained in the model accounted for 34% of the variance. It seems to
reflect Caregiver-Child Mutual Engagement and Relationship, including mutual
responsiveness and positive affect and affection (recall negatively phrased items are reverse
scored and a positive loading reflects a lack of those negative behaviors). Note that child items
are also loaded on this factor, which suggests true caregiver and child reciprocal interaction,
engagement, and relationship. Low scores on this factor may reflect caregiver-child detachment

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 8

and very little meaningful warm, sensitive, responsive interaction, which is commonly
observed in these orphanages.
NIH-PA Author Manuscript

The second factor, which accounted for 16.2% of the variance, reflects Caregiver
Punitiveness, including both caregiver and child negative affect. The reverse scoring reflects
the absence of these behaviors, so low scores would be associated with punitiveness while high
scores would signal the absence of punitiveness (which often characterizes orphanage
caregivers, at least while being observed by strangers).

The third factor, accounting for 10.2% of the variance, seems to represent Caregiver- vs. Child-
Directed Behaviors and Intrusiveness (again the reverse scoring makes the factor positive so
high scores represent many child-directed interactions and the absence of caregiver
punitiveness). The structure of this factor is less simple than that of the first two, because three
of the four items that defined it have moderate loadings (greater than .3) on the other two
factors. This pattern seems explicable, however. Caregiver-directed behaviors, for example,
are often displayed in orphanages at the expense of child-directed behaviors, they may be
intrusive, and they may be part of disciplinary situations. Further, caregiver intrusiveness also
occurs during disciplinary and punishment activities (Factor 2), and child-directed behaviors
should occur more frequently in mutual engagement (Factor 1).

A factor analysis of the pooled sample from both St. Petersburg orphanages was also conducted
NIH-PA Author Manuscript

after the orphanage item mean was subtracted from each item's rating (which removed any
influence of orphanage mean differences from the correlation matrix). The results were quite
similar to that reported above for the Managua orphanage, except the Managua third factor
was now the second factor and the items “15. Child responsiveness and anticipation,” and “14.
Child detachment and failure to respond” loaded on the Child- vs. Caregiver-Directed
Behaviors and Intrusiveness factor to a greater extent than on the first factor. Given the
inclusion of a very different orphanage with much improved caregiving in the St. Petersburg
data, this degree of factorial consistency is encouraging.

Thus, the CCSERRS primarily reflects Caregiver-Child Mutual Engagement and Relationship,
with subordinate components of Caregiver Punitiveness and Caregiver- vs. Child-Directed
Behaviors and Intrusiveness. It should be noted that the caregiving in the Nicaragua orphanage
that is the basis of this factor analysis was quite minimal, with a Total Score of 1.16 on a scale
ranging from 0 to 3.0 (an average very close to the 1.06 obtained for the no-intervention
orphanage in St. Petersburg), so the factor structure could change if caregiver behavior was
more variable and ratings were spread more evenly throughout the 0-3 range (although the
similarity in factor structure between the Nicaraguan and St. Petersburg data suggests the
change might be minimal). The low levels of punitiveness and negative affect might also
NIH-PA Author Manuscript

influence the factor structure, which was true in both samples.

Discussion
This paper reports preliminary reliability, validity, and psychometric information on a newly
developed rating scale initially created to characterize caregiver-child social-emotional
interactions and relationships in orphanages but that could also be used in non-residential early
care and education settings and potentially parent-child interactions in a home environment.
The intent was to identify a few dimensions of caregiver-infant/toddler/young child interactions
that comprehensively covered the range of specific characteristics of caregiver-child social-
emotional interactions and relationships, and to construct a scale that could be administered in
a relatively short period of time and would not require extensive coder training, manuals, or
materials.

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 9

The preliminary data presented on the Caregiver-Child Social/Emotional Relationship Rating

Scale (CCSERRS) tentatively suggests that it can be reliably administered even using
observation periods of as little as 5 or as long as 30 minutes spread over three kinds of caregiver
NIH-PA Author Manuscript

activities (feeding, dressing/bathing, free play). Reliability was replicated over seven different
coders working in three different orphanages. Brief training can prepare several coders (five
in this study) to rate caregivers in a reliable and consistent fashion. Moreover, variability in
scores was associated with individual differences in caregivers to a much greater extent than
in coders (38% to 1%), and ratings of caregivers were similar across different types of
caregiving activities (i.e., feeding, dressing/bathing, free play) and for caregivers attending
children birth to 4 and 4 to 8 years of age.

The CCSERRS primarily reflects Caregiver-Child Mutual Engagement and Relationship with
subordinate components of Caregiver Punitiveness and Caregiver- vs. Child-Directed
Behaviors and Intrusiveness.

A pilot validity study demonstrated that most of the 18 items and the Total Score significantly
discriminated between caregivers who received a special intervention of training and structural
changes designed to improve their warm, caring, sensitive, and responsive interaction with
children vs. orphanage caregivers who did not. This suggests that the CCSERRS likely has
construct validity for reflecting positive social/emotional interactions and relationships
between caregivers and children.
NIH-PA Author Manuscript

The CCSERRS has the advantage of being exclusively directed at caregiver-infant/toddler/

young child social-emotional interactions and relationships, and it measures the extent of
engagement, responsiveness, positive versus negative affect and affection, child- vs. caregiver-
directed interactions, and support. These characteristics are only a small part of the more
commonly used comprehensive home and group care environmental rating scales (i.e., HOME,
ITERS/ECERS). Since these latter scales characterize the total physical and behavioral
environment, the CCSERRS could be used in addition to the HOME or ECERS if a special
focus on the social/emotional interaction is needed.

The authors encourage colleagues to try the CCSERRS and communicate the details of is use,
reliability, validity, and psychometric properties to the authors. In this way a body of
psychometric information can be amassed that can help overcome the limits of the present
report (e.g., small Ns, only orphanage contexts, limited observation periods, specific caregiver
activities, non-standardized observation procedures).

Clinical Implications
Early caregiver-child social-emotional interactions and relationships are theorized to have
NIH-PA Author Manuscript

major implications for infant-toddler and longer-term mental health and behavioral
competence, and the empirical literature generally confirms this principle (see above). A
relatively simple scale that more directly rates such interactions has the potential of facilitating
research and practice in this domain.

Appendix 1. The Caregiver-Child Social/Emotional and Relationship Rating

Scale (CCSERRS)
Constructs to be Rated
Caregiver Engagement
1. Caregiver detachment—in the presence of a child or children, caregiver does not make
eye contact; does not talk to children in a social way; does not engage children when that would

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 10

be possible; ignores them when they are eating, dressing, playing; does other tasks during free
time, such as talk to other caregivers, takes a break to smoke or groom herself; props bottle
instead of feeding child.
NIH-PA Author Manuscript

2. Caregiver failure to respond—caregiver does not respond to child's vocalizations,

smile, other social initiative, distress (crying, frustration); caregiver responds but in a non-
social, business-like, mechanical manner. Caregiver does not psychologically engage the child.
Caregiver fails to share enthusiasm, joy, or excitement of children; caregiver fails to display
pride in children's accomplishment. If there is no stimulus from child, this is rated high (i.e.,
failure to respond is frequent).

3. Caregiver psychological availability and receptivity to children—caregiver looks

child in the eye and holds child face-to-face while feeding or in play; caregiver listens
attentively when children speak (e.g., looks at them, responds by nodding); caregiver gets at
the level of the child (kneels, sits on floor with children) to have better eye contact, to hear a
child, to talk or play with children; caregiver attends to and engages appropriately a child who
is constructively engaged (she goes with the child's flow).

4. Caregiver responds to child—caregiver paces caregiving to match child's behavior

(e.g., waits to give next spoonful of food until the child is ready), is patient with slow eaters
or dressers; tolerates messiness; caregiver responds to child's behavior, signals, and
NIH-PA Author Manuscript

communication attempts; caregiver makes empathetic verbalizations or facial expressions to

child's actions and events; caregiver shares joy and excitement, pride in children's
accomplishments; caregiver imitates child's behavior (vocalization, smile, actions); caregiver
praises, encourages, rewards child for child's behavior. Mechanical response is scored low.

Caregiver/Child-Directed Behaviors
5. Caregiver directed behaviors—caregiver directs child physically or verbally in a non-
disciplinary activity (e.g., how to eat, get dressed, wash, play) and expects the children to follow
her directions or imitate her actions; caregiver talks at children; caregiver models use of an
object for child to imitate in play or shows the child what to do (with object, how to dance);
caregiver labels objects, parts of body, food, and expects imitation or no response; caregiver
reads books to children with no child participation; caregiver teaches children about something,
explains cause-effect, gives information without expectation of questions or discussion.

6. Child-directed behaviors—caregiver lets child lead and she follows (e.g., allowing the
child to do something to the caregiver such as pulling hair or touching glasses) and caregiver
responds in playful or positive way; caregiver engages in conversation back and forth (even if
NIH-PA Author Manuscript

child responds non-verbally); caregiver plays reciprocal games with children (roll ball back
and forth, play catch, peek-a-boo, goochie-goo); caregiver asks what the child wants to do and
how the child wants to do it; caregiver asks open-ended questions; caregiver promotes peer
interaction (turn taking, conversation, cooperation, sharing, pretend play). Caregiver
appropriately allows children to “do their own thing.”

Behavioral Control
7. Caregiver intrusiveness—caregiver performs a caregiving action (feeding, bathing,
changing) without engaging or preparing the child (“ready or not”); she catches the child by
surprise; caregiver hurries caregiving without waiting for child to adjust (e.g., keeps spooning
food into mouth, keeps putting bottle in mouth, keeps soaping child when child is upset).
Caregiver stimulated child when child does not pay attention or respond to it. Caregiver intrudes
on children who are appropriately engaged and prohibits what they want to do or redirects the
child when it is unnecessary, she interrupts child activity to get the child to do something else.

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 11

8. Caregiver behavioral control/obedience/discipline—caregiver directs children

individually or in group behavior (stand in line, sit against the wall, hold onto another child)
and expects conformity to caregiver intentions; caregiver “corrects” behavior (e.g., keep in
NIH-PA Author Manuscript

line, go over there, no food fights, etc.). Child is interested in one thing and caregiver reorients
child to another.

9. Caregiver punishes a child—caregiver physically and negatively grabs a child to move

him or her; corrects child in a clearly negative way; hits or spanks a child. Verbally abuses,
yells or talks down to child in a disrespectful manner.

10. Caregiver support, empathy, guidance—caregiver kindly and positively

encourages child to feed, dress, bath self; caregiver is sympathetic or empathetic with a child
having difficulty or one who is hurt and crying; caregiver changes child behavior in a positive
and supportive way by providing “guidance” or “suggestions;” caregiver explains a rule or
consequences and calmly redirects behavior all in a positive and supportive way.

Caregiver Affect with Children

11. Caregiver negative affect—caregiver displays negative affect to a child (frowns,
scowls); caregiver speaks negatively, harshly, with annoyance, hostility, and scolds or shouts
at the child; caregiver is critical of children, puts them down, uses sarcasm, blames child, says
“bad boy”.
NIH-PA Author Manuscript

12. Caregiver displays positive affect and affection—she smiles at child; talks to the
child in a positive, warm, affectionate, supportive way; she hugs, kisses, and warmly holds a
child.

13. Caregiver animation and expressiveness—caregiver talks or reads with animation

and expressiveness; is animated or expressive when engaged with children; changes emotional
expression to fit moment-to-moment activity both positive and negative (e.g., winces in
animated pain when child falls, makes empathetic smiles or frowns to the child's actions);
expresses positive emotion and enthusiasm to a child's accomplishments (praises and smiles
to child's drawing or stack of blocks).

Child Engagement
14. Child detachment and failure to respond to caregiver—child seems unresponsive
emotionally and physically to appropriate and positive caregiver behavior and caregiver
attempts to engage the child (e.g., attempts to get the child to smile or laugh); child seems
unresponsive emotionally to negative caregiver behavior or “discipline” or harsh words; child
NIH-PA Author Manuscript

does not smile or “brighten,” get “excited,” wiggle with anticipation, or raise arms to be picked
up when a caregiver comes over to attend to them or pick them up. If child responds in a forced,
non-social, mechanical manner score high.

15. Child responsiveness and anticipation—child readily makes eye contact with
caregiver, attempts to engage caregiver, asks questions or talks to caregiver, calls for caregiver;
child anticipates caregiver engagement by smiling, “brightening,” shows excitement or
wiggles, raises arms to be picked up when caregiver attends to him or her; child responds to
caregiver attempts to engage the child by talking back, playing peek-a-boo; child anticipates
comfort or help from caregiver when crying, frustrated, assaulted by other children or having
toy taken away; child initiates talk to caregiver and clearly expects caregiver to talk back. If
caregiver provides no stimulus to child, score 0.

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 12

Child Affect
16. Child negative affect—child cries or whimpers, shows anger or other negative affect.
NIH-PA Author Manuscript

17. Children's positive affect—child displays positive affect (smile, laugh) in interaction
with caregiver during routine caregiving or play; child is “alive” and emotionally labile,
matching emotions to caregiver's emotions; child smiles or laughs when tickled. Child is happy,
smiling, laughing on own or with peers.

Child Relationship
18. Child relationship with caregivers—infant watches caregiver intently; infant follows
with eyes caregivers deliberate movements. Child uses caregiver as a “secure base” by sharing
a toy with her, asking a question, looking at the caregiver to share a positive or negative
experience; child seeks caregiver when experiencing stress or is upset; child checks in with
caregiver from a distance while playing (e.g., makes eye contact or vocal or verbal contact with
caregiver); child shows separation anxiety or distress when caregiver leaves or turns head away,
child attempts to re-engage caregiver; child shows wariness of strangers and may cling to
caregiver or hold on to skirt.

Scoring
NIH-PA Author Manuscript

Each item is scored on a four-point scale (0=never, 1=rarely, 2=frequently, 3=consistently)

reflecting the frequency with which the named behavior (positive or negative) occurred relative
to the potential opportunities for it to occur. Note special scoring for items 2, 4, 14, 15 (see
above). Items 1, 2, 5, 7-9, 11, 14, and 16 are negative or undesirable but scoring is high if they
are consistently present; these items must be reverse scored before calculating total or subscale
scores.

References
Arnett J. Caregivers in day care centres: Does training matter? Journal of Applied Developmental
Psychology 1989;10:541–552.
Aviezer O, Sagi A, Resnick G, Gini M. School competence in young adolescence: Links to early
attachment relationships beyond concurrent self-perceived competence and representations of
relationships. International Journal of Behavioral Development 2002;26:397–409.
Blizzard, RM. Psychosocial short stature. In: Lifshitz, F., editor. Pediatric endocrinology. New York,
NY: Marcel Dekker; 1990. p. 77-91.
Bradley RH, Caldwell BM. Caregiving and the regulation of child growth and development: Describing
proximal aspects of caregiving systems. Developmental Review 1995;15:38–85.
NIH-PA Author Manuscript

Caldwell, BM.; Bradley, RH. Home Observation for Measurement of the Environment. Little Rock:
University of Arkansas at Little Rock; 1984.
Carlson EA. A prospective longitudinal study of disorganized/disoriented attachment. Child
Development 1998;69:1107–1128. [PubMed: 9768489]
DeWolff MS, van IJzendoorn MH. Sensitivity and attachment: A meta-analysis on parental antecedents
of infant attachment. Child Development 1997;68:571–591. [PubMed: 9306636]
Edwards CP, Raikes H. Extending the dance: Relationship-based approaches to infant/toddler care and
education. Young Children 2002;57:10–17.
Erickson, MF.; Sroufe, LA.; Egeland, B. The relationship between quality of attachment and behavior
problems in preschool in a high-risk sample. In: Bretherton, I.; Waters, E., editors. Growing points of
attachment Theory and Research. Monographs of the Society for Research in child Development. Vol.
50. 1985. p. 147-166.Serial #209
Fonagy, P.; Target, M.; Steele, M.; Steele, H.; Leigh, T.; Levinson, A.; Kennedy, R. Crime and
attachment: Morality, disruptive behavior, borderline personality disorder, crime, and their

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 13

relationships to security of attachment. In: Atkinson, L.; Zucker, K., editors. Attachment and
psychopathology. New York: Guilford Press; 1997. p. 223-274.
Goossens C, Melhuish EC. On the ecological validity of measuring the sensitivity of professional
NIH-PA Author Manuscript

caregivers: The laboratory versus the nursery. European Journal of Psychology of Education
1996;11:169–176.
Greenberg, MT. Attachment and psychopathology in childhood. In: Cassidy, J.; Shaver, PR., editors.
Handbook of attachment: Theory, research, and clinical applications. New York: Guilford; 1999. p.
469-496.
Groark, CJ.; McCall, RB.; and the Whole Child Inernational Team. Managua orphanage baseline report.
Pittsburgh, PA: University of Pittsburgh Office of Child Development; 2007.
Groark CJ, Muhamedrahimov RJ, Palmov OI, Nikiforova NV, McCall RB. Improvements in early care
in Russian orphanages and their relationship to observed behaviors. Infant Mental Health Journal
2005;26:96–109.
Gunnar, M. Effects of early deprivation; Findings from orphanage-reared infants and children. In: Nelson,
CA.; Luciana, M., editors. Handbook of developmental cognitive neuroscience. Cambridge, MA:
MIT Press; 2001. p. 617-629.
Harms, T.; Clifford, RM.; Cryer, D. Early Childhood Environmental Rating Scale. Revised. NY: Teachers
College Press; 1998.
Harms, T.; Cryer, D.; Clifford, RM. Infant/Toddler Childhood Environmental Rating Scale. NY: Teachers
College Press; 1990.
Howes C, Stewart P. Child's play with adults, toys, and peers: An examination of family and child care
NIH-PA Author Manuscript

influences. Developmental Psychology 1987;23:423–430.

Johnson, DE. Medical and developmental sequelae of early childhood institutionalization in Eastern
European adoptees. In: Nelson, CA., editor. The effects of early adversity on neurobehavioral
development. Vol. 31. Mahwah, NJ: Lawrence Erlbaum; 2000. p. 113-162.
Kontos, S.; Howes, C.; Shinn, M.; Galinsky, E. Quality in family child care and relative care. New York:
Teachers College Press; 1995.
Koren-Karie N, Sagi-Schwartz A, Egoz-Mizrachi N. The emotional quality of child care centers in Israel:
The Haifa study of early child care. Infant Mental Health Journal 2005;26:110–126.
Landry SH, Smith KE, Miller-Loncar CL, Swank PR. Predicting cognitive-language and social growth
curves from early maternal behaviors in children at varying degrees of biological risk. Developmental
Psychology 1997;33(6):1040–1053. [PubMed: 9383626]
Landry SH, Smith KE, Swank PR. Responsive parenting: Establishing early foundations for social,
communication, and independent problem-solving skills. Developmental Psychology 2006;42:627–
642. [PubMed: 16802896]
Landry SH, Smith KE, Swank PR, Miller-Loncar CL. Early maternal and child influences on children's
later independent cognitive and social functioning. Child Development 2000;71:358–375. [PubMed:
10834470]
Lyons-Ruth K, Alpern L, Repacholi D. Disorganized infant attachment classification and maternal
NIH-PA Author Manuscript

psychosocial problems as predictors of hostile-aggressive behavior in the preschool classroom. Child

Development 1993;64:572–585. [PubMed: 8477635]
MacLean K. The impact of institutionalization on child development. Development and Psychopathology
2003;15:853–884. [PubMed: 14984130]
McGuire J, Richman N. Management of behavior problems in day nurseries [Special Issue]. Early Child
Development and Care 1987;45
Melhuish EC. The quest for quality in early day care and preschool experience continues. International
Journal of Behavioral Development 2001;25:1–6.
Melhuish EC, Mooney A, Martin S, Lloyd E. Type of child care at 18 months: I. Differences in
interactional experiences. Journal of Child Psychology and Psychiatry 1990;31:849–859. [PubMed:
2246337]
Muhamedrahimov RJ, Palmov OI, Nikiforova NV, Groark CJ, McCall RB. Institution-based early
intervention program. Infant Mental Health Journal 2004;25:488–501.

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 14

Nelson CA, Zeanah CH, Fox NA, Marshall PJ, Smyke AT, Guthrie D. Cognitive Recovery in socially
deprived young children: The Bucharest early intervention project. Science Dec 21;2007 318:1937–
1940. [PubMed: 18096809]
NIH-PA Author Manuscript

Pierrehumbert B, Ramstein T, Krucher R, El-Najar S, Lamb ME, Halfon O. L'evaluation du lieu de vie
du jeune enfant: Développement d'un instrument. Bulletin de Psychologie, XLIX 1996;426:565–584.
Rosas, J.; McCall, RB. Characteristics of institutions, interventions, and children's development.
Pittsburgh, PA: University of Pittsburgh Office of Child Development; 2008.
Rothbaum F, Weisz JR. Parental care giving and child externalizing behavior in non-clinical samples: A
meta-analysis. Psychological Bulletin 1994;116:55–74. [PubMed: 8078975]
Shaw DS, Owens EB, Vondra JI, Keenan K, Winslow EB. Early risk factors and pathways in the
development of early disruptive behavior problems. Development and Psychopathology 1997;8:679–
700.
St. Petersburg – USA Orphanage Research Team. Characteristics of children, caregivers, and orphanages
for young children in St. Petersburg, Russian Federation. Journal of Applied Developmental
Psychology: Child Abandonment, Special Issue 2005;26:477–506.
St. Petersburg-USA Orphanage Research Team. The effects of early social-emotional and relationship
experience on the development of young orphanage children. Society for Research in Child
Development Monograph 2008;73(3) Serial No. 291.
Steelman LM, Assel MA, Swank PR, Smith KE, Landry SH. Early maternal warm responsiveness as a
predictor of child social skills: Direct and indirect paths of influence over time. Applied
Developmental Psychology 2002;23:135–156.
NIH-PA Author Manuscript

Sylva, K. Target child observation. In: Sylva, K.; Stevenson, J., editors. Social behavior and competence
in childhood. Windsor, UK: NFER-Nelson; 1997.
van IJzendoorn, MH.; Sagi, A. Cross-cultural patterns of attachment. In: Cassidy, J.; Shaver, PR., editors.
Handbook of attachment. New York, NY: Guilford Press; 1999. p. 713-734.
van IJzendoorn MH, Tevecchio LWC. Quality of center day care and attunement between parents and
caregivers. Center day care in cross-national perspective. Journal of Genetic Psychology
1998;159:437–456. [PubMed: 9845974]
Zeanah, CH.; Smyke, AT.; Koga, SFM. Effects of foster care following institutionalization on cognitive
development and behavior problems. In: Nelson, CA., (Chair), editor. The effects of early
institutionalization on brain behavior development: The Bucharest Early Intervention Project;
Symposium presented at the biennial meeting of the Society for Research in Child Development;
Tampa, FL. 2003 Apr.
NIH-PA Author Manuscript

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 15

Table 1
Comparison of Item Ratings for Non-Intervention vs. Intervention
NIH-PA Author Manuscript

St. Petersburg Orphanages

Item NoI (N=17) T+S (N=36) t(df=42) eta2
M (SD) M (SD)

1. Cgr. detachment .71(.77) 2.78(.42) 12.67*** .76

2. Cgr. failure to respond .53(.80) 2.64(.54) 11.30*** .71

3. Cgr. availability, receptivity 1.00(.79) 2.78(.49) 10.11*** .68

4. Cgr. responds to child .82(.81) 2.34(.58) 7.78*** .54

5. Cgr.-directed behaviors .29(.59) 1.25(.65) 5.15*** .34

6. Child-directed behaviors .24(.44) 1.81(.71) 8.38*** .58

■ 7. Cgr. intrusiveness .53(.62) 2.50(.70) 9.92*** .66

■ 8. Cgr. control, obedience, discipline 2.35(1.00) 2.61(.55) 1.22 .03

■ 9, Cgr. punishes child 3.00(.00) 2.97(.17) 0.68 .01
10. Cgr. support, empathy, guidance .82(.81) 2.28(.70) 6.71*** .47

■ 11. Cgr. negative affect 3.00(.00) 2.97(.17) 0.68 .01

NIH-PA Author Manuscript

12. Cgr. positive affect, affection .65(.79) 2.08(.73) 6.51*** .45

13. Cgr. animated, expressive .82(.95) 2.22(.80) 5.60*** .38

■ 14. Child detachment, failure to respond .41(.62) 2.42(.87) 8.49*** .59

15. Child responsiveness, anticipation .47(.62) 2.06(.86) 6.79*** .48

16. Child negative affect 2.65(.61) 2.89(.32) 1.90 .07

17. Child positive affect .35(.79) 1.78(.87) 5.75*** .39

18. Child relationship w/cgr. .41(.62) 2.44(.56) 11.96*** .74

Total Score (sum) 19.06(7.01) 43.03(5.52) 13.29*** .78

2.39(.31) 13.29*** .78

Items that were reverse scored so high values represent “good” behavior for each item.
*
p≤.05
**
p≤.01
NIH-PA Author Manuscript

***
p≤.001, respectively.

Eta2 is the estimate of variance associated with the difference between orphanages.

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.
 McCall et al. Page 16

Table 2
Factor Analysis of the 18 Coding Items
NIH-PA Author Manuscript

Factors
1 2 3

Caregiver-Child Mutual Positive Engagement

18. Child relationship with caregivers 0.80
4. Caregiver responds to child 0.79
10. Caregiver support, empathy, guidance 0.78
1. Caregiver detachment 0.76
12. Caregiver display positive affect/affection 0.75
3. Caregiver availability and receptivity 0.75
13. Caregiver animated and expressive 0.73
17. Children's positive affect 0.71
15. Child responsiveness and anticipation 0.64
14. Child detachment and failure to respond 0.62
2. Caregiver failure to respond 0.62 0.30
Caregiver Punitiveness
NIH-PA Author Manuscript

9. Caregiver punishes a child 0.81

11. Caregiver negative affect 0.81
16. Child negative affect 0.74
Child-Caregiver Directed Interaction
8. Caregiver behavioral control/obedience/discipline 0.61 0.35
5. Caregiver-directed behaviors 0.87
6. Child-directed behaviors 0.40 0.68
7. Caregiver intrusiveness 0.49 0.57

Percent of variance 34.0 16.2 10.2

Notes: Principal Component Analysis with Varimax rotation. Only coefficients above 0.30 are shown. The model used items “reverse coded” where
necessary so that high values are consistently “good.” Among the data used are different codings on the same caregiver, so not all observations are
independent.
NIH-PA Author Manuscript

Infant Ment Health J. Author manuscript; available in PMC 2010 June 15.

View publication stats