Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 DOI 10.1007/s10488-007-0121-3 ORIGINAL PAPER Implementation of Evidence-based Practice in Child Welfare: Service Provider Perspectives Gregory A. Aarons Æ Lawrence A. Palinkas Published online: 5 April 2007 Ó Springer Science+Business Media, LLC 2007 Abstract Implementation of evidence-based practices (EBP) in child welfare is a complex process that is often fraught with unanticipated events, conflicts, and resolutions. To some extent, the nature of the process, problems, and solutions may be dependent on the perspectives and experiences of a given stakeholder group. In order to better understand the implementation process in the child-welfare system, we interviewed comprehensive home-based services (CHBS) case managers who were actively engaged in implementing an EBP to reduce child neglect in a state youth services system. Six primary factors were identified as critical determinants of EBP implementation: (1) Acceptability of the EBP to the caseworker and to the family, (2) Suitability of the EBP to the needs of the family, (3) Caseworker motivations for using the EBP, (4) Experiences with being trained in the EBP, (5) Extent of organizational support for EBP implementation, and (6) Impact of EBP on process and outcome of services. These factors reflect two broader themes of attitudes toward or assessments of the EBP itself and experiences with learning and delivering the EBP. Eventual implementation is viewed as the consequence of perseverance, experience, and flexibility. G. A. Aarons (&) Child & Adolescent Services Research Center, University of California, San Diego, 3020 Children’s Way, MC-5033, San Diego 92123-4282 CA, USA e-mail: gaarons@ucsd.edu L. A. Palinkas Professor of Social Work, Anthropology and Preventive Medicine, School of Social Work MRF 339, University of Southern California, Los Angeles, CA 90089-0411, USA e-mail: palinkas@usc.edu Keywords Implementation
Evidence-based practice
Child-welfare Introduction Repeated demonstration of the beneficial effects of evidence-based psychotherapeutic, case-management, and pharmacologic interventions has not led to widespread implementation of such interventions in usual care settings (Jensen, 2003; Rotheram-Borus & Duan, 2003). Service providers often rely upon non-evidence based practices in providing services to children and families (Bickman, Heflinger, Lambert, & Summerfelt, 1996). This gap between usual care and evidence-based practice (EBP) is characteristic of mental health services in general and has been attributed to a number of factors including provider attitudes toward adopting evidence-based practices (Aarons, 2004, 2006; Aarons & Sawitzky, 2006), time and resources of practitioners, insufficient training, lack of access to peer-reviewed research journals, lack of feedback and incentives for use of EBPs, flawed logic and assumptions behind the design of efficacy and effectiveness research trials that fail to consider the complexity of realworld service settings, and inadequate infrastructure and systems to support translation of EBP for real world settings (Glasgow, Lichtenstein, & Marcus, 2003; National Institute of Mental Health, 1999; Schoenwald & Hoagwood, 2001). There is a critical gap in our understanding of barriers and facilitators of EBP implementation (Aarons, 2005; Burns, Hoagwood, & Mrazek, 1999; Garland, Kruse, & Aarons, 2003; Glisson, 1992; Glisson, 2002; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001) and little literature exists on service provider perspectives on 123 412 actually implementing EBPs. The study of implementation is also vitally important in determining both the effectiveness and sustainability of EBPs in real world settings. For example, effectiveness of an EBP will likely be compromised if it is poorly implemented (Henggeler, Pickrel, & Brondino, 1999) during the process of local adaptation (Elliott & Mihalic, 2004). Poor implementation could lead to a negative appraisal of EBP effectiveness when it is failure of the implementation process rather than the EBP that is responsible for poor outcomes. A recent qualitative study in California with multiple stakeholders suggests that EBPs found to be efficacious in controlled settings are perceived as impractical or difficult to apply in real world settings (Hurlburt & Knapp, 2003). The ability to understand the EBP implementation process (how something happens) and implementation result (effect on the system, providers, consumers) is critical to determining the likelihood that such practices will be tenable in complex real world settings. Child welfare systems present unique challenges to EBP implementation in terms of the structure, processes, workers, and service population. Child welfare systems are typically highly bureaucratic in nature, and a high degree of bureaucracy has been linked to poor service worker attitudes toward adopting evidence-based practices (Aarons, 2004). Further, there are a number of common process factors (training, communication, supervision) that take place across service sectors and types of services. In child welfare, services must focus first on the protection and well-being of youths; however, youth care is mediated through parents and caregivers who may or may not be amenable to receiving such services. There is a high degree of variability in clients in regard to age of parents, parent education level and cognitive ability, parent engagement in services, age of children, and number of children in the home. Thus, implementation may be impacted by system, structural, process, and person factors. However, there has been little empirical research to date on barriers and facilitators to implementation of evidence-based practice in child welfare, and even less that highlights the perspectives of service providers on EBP implementation. The purpose of the present study was to elucidate direct service provider’s perspectives regarding factors that influence implementation of an EBP in a child-welfare system and to understand which factors might be modifiable in order to facilitate implementation of EBPs. The context is a statewide implementation of SafeCare (SC) (Gershater-Molko, Lutzker, & Wesch, 2003), an intervention designed to reduce child neglect among at-risk parents. ‘‘At-risk’’ is defined as having a report of abuse or neglect filed with the state Office of Children’s Services. Training of case managers in SC begins with five full-day interactive training sessions. Consistent with the behavioral skill 123 Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 orientation of the model, training involves a minimum of didactic presentation and utilizes modeling of provider skills by trainers, role plays of skills, and assessment of initial skill acquisition with checklists. Training materials, a training manual and a workbook have been developed and used in training SC providers in the ongoing study. The training model for SC is described in detail in Filene, Lutzker, Hecht, and Silovsky (2005). During the second phase of training and after completing the five full-day sessions, the implementation of SC by case managers is observed and coached by ongoing consultants (i.e., trainers) in the actual practice setting. Methods Participants Participants in this study were fifteen case managers and two ongoing consultants involved in the implementation of SC and ongoing fidelity monitoring of the intervention. Case manager participants were selected by maximum variation sampling to represent those having the most positive and those having the most negative views of SC based on results of a web-based quantitative survey asking about the perceived value and usefulness of SC. At the time this study was conducted, there were two ongoing consultants who were both included as interviewees. Demographic data is not presented in order to assure confidentiality. Semi-structured interviews were conducted over a twoweek period by an experienced doctoral level medical anthropologist. An interview guide was designed to elucidate the experience of being trained and using SC with a focus on identifying barriers and facilitators to implementation. However, not every informant provided information on every topic since the intention was to allow informants to elaborate or focus on issues he or she considered to be the most important and on which he or she had an opinion. Interview duration was approximately one hour. Data Management and Analysis All interviews were digitally recorded and transcribed by a professional transcriber. Transcriptions were reviewed and checked for accuracy by at least one of the authors or a research assistant. Using a methodology of ‘‘Coding Consensus, Co-occurrence, and Comparison’’ outlined by Willms et al. (1992) and rooted in grounded theory (i.e., theory derived from data and then illustrated by characteristic examples of data) (Glaser & Strauss, 1967), interview transcripts were analyzed in the following manner. First, the empirical material contained in the interviews Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 was independently coded by the project investigators to condense the data into analyzable units. Segments of text ranging from a phrase to several paragraphs were assigned codes based on a priori (i.e., from the interview guide) or emergent themes. Three randomly selected complete transcripts were independently coded by each investigator. Disagreements in assignment or description of codes were resolved through discussion between investigators and enhanced definition of codes. The final list of codes, constructed through a consensus of research team members, consisted of a numbered list of themes, issues, accounts of behaviors, and opinions that related to organizational and system characteristics that influence implementation of SC. All transcripts were then coded by a trained research assistant who met regularly with the first author to discuss progress and resolve any coding discrepancies. With the final coding structure, each investigator separately reviewed transcripts to determine level of agreement in the codes applied. Second, based on these codes, the computer program QSR NVivo (Fraser, 2000) was used to generate a series of categories arranged in a treelike structure connecting text segments grouped into separate categories or ‘‘nodes.’’ These nodes and trees were used to examine the association between different a priori and emergent categories, and to identify the existence of new, previously unrecognized categories. Third, through the process of constantly comparing these categories with each other, the different categories were further condensed into broad themes using a format that places EBP implementation within the framework of organizational and system characteristics (Glaser & Strauss, 1967). 413 positive statements regarding the comprehensive nature of the EBP as evidenced by the statement ‘‘...it has all the developmental information and everything. And I’m the kind of person that likes that step-by-step type of thing telling me what to do...’’ For many caseworkers the structured approach was viewed as a positive aspect of the model; however, some very experienced caseworkers reported that the structured approach was not needed because the issues were already addressed as part of their interaction with their clients. Acceptability to clients also focused on the content, which was positively perceived. For instance, ‘‘...I’ve had clients say that the health manual, for example, was very helpful.... The Home Cleanliness, it opened their eyes in a lot of ways, the services and the products that we provide to increase home safety for the children...’’ The structured approach also helped to frame the model as an aid and not stigmatizing, although case managers had to frame it appropriately for clients. Case managers also found the intervention useful in facilitating communications with a diverse range of families, ‘‘...I just have to say [to the family], ‘Well we never know what kind of family that we’re going to run into and how much they will understand. And it’s [the EBP] all made the same.’ So they’re like ‘Okay’.’’ However, there were some clear negative reactions and resistance to the EBP model. Some of those interviewed expressed some resentment with the suggestion that there may be more effective ways to provide services and that their good faith efforts to bring about positive change with their clients might be less than optimal. For example one case manager stated: ‘‘I would rather sacrifice the [EBP] being perfect than sacrifice the rapport that I have with my clients with their other bigger issues.’’ Results Six primary factors emerged as critical determinants of EBP implementation in this study: (1) Acceptability of the EBP to the caseworker and to the family, (2) Suitability of the EBP to the needs of the family, (3) Caseworker motivations for using the EBP, (4) Experiences with being trained in the EBP, (5) Extent of organizational support for EBP implementation, and (6) Impact of the EBP on the process and outcome of services. Each of these factors is addressed below. Acceptability of EBP to the Caseworker and to the Family Caseworkers generally reported a positive evaluation of the content of the EBP. This is illustrated by the statement that ‘‘...every time we do ...certain modules, especially the parenting, I mean the help is good for anybody. So that’s really informative, even for me...’’ There were also Suitability of EBP to the Needs of the Family The perception of suitability to the needs of the family was seen as important and arose in the context of a number of issues. For instance, caseworkers perceived SC to be more appropriate and effective with families where the child was recently returned to the home after having been removed (i.e., reunification case) than in families where the children have not been removed and participation in caseworker involvement is voluntary (i.e., prevention case). According to one of the caseworkers interviewed, ‘‘Reunification cases, they’ve already messed up.... and they’re really working towards wanting to get better.... So when we come into the home, they’re just excited that we’re there....Voluntary, they really haven’t gotten in trouble yet and...they just don’t want to cooperate very well...’’ Voluntary cases are also called ‘‘prevention’’ cases where clients may not have an actual charge filed. Still, a court mandate and in- 123 414 creased client motivation appear to aid the caseworker with implementation of the model. The age of parents and children was seen as an important determinant of perceived suitability of the intervention. Caseworkers were generally of the opinion that the SC intervention was less appropriate for parents with older children (e.g., 11 years of age or older). This issue was raised by several of the caseworkers who commented on the difficulty of implementing the intervention in households with older children. For example, ‘‘...Playtime is like, I mean it’s not like you can sit there and watch a mom hold a teddy bear in front of her baby and say, ‘‘goo-goo, gah-gah.’’ You know? My kids [i.e., clients] are like 10 and 11.... At this point I just feel like what part of the Parent Activity Training can I do? I can’t really do play time. I can’t do bath time. I can’t do dress time...’’ Instead, some of the caseworkers requested more information on parenting teens. As was noted by one case manager, ‘‘...the families that it’s not working with are the families that have older kids.... And it’s like I start doing that training and they’re just like yeah, yeah, no, no.... They’re just like ‘this is ridiculous’.’’ This situation presents a quandary for case managers in that there may be reduced credibility with clients where the intervention is clearly a poor fit - but it is still required to be used. Complexity of family problems also influenced the perceived suitability of the EBP to the family’s and client’s needs. Some caseworkers noted that the complex nature of family problems and situations limited the appropriateness and effectiveness of the EBP. As described by one caseworker, ‘‘...we do run into problems sometimes where we may go into the family, the families’ homes and they have other issues. I mean we have to find a job immediately or we have to find housing immediately and so we’re addressing those things first. Or they’re in jail or they got thrown in jail, so we’re addressing those issues first and then we go. I’ve had a couple of cases where I can’t even touch on the [EBP] stuff until we get everything else straightened out...’’ Thus, the sequencing of an EBP in an array of services may be an important concern because of other competing demands. For the case manager, the lack of fit between the EBP and the family could cause ambiguity and discomfort in providing services. For example, one case manager noted that ‘‘...I do [the EBP] because I have to do it. And with some families that’s the only reason I will do it... Because like I told you, some of the families I just feel like they’re very high functioning...’’ And once again the fit of the model with family characteristics was a concern: ‘‘...But there’s one family that I can think of that has like a 3-year old and the rest of my families have 9 and 10-year olds, 16-year olds, 15-year olds.’’ 123 Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 Caseworker Motivations for Using the EBP There were a number of motivations that impacted implementation of the EBP including enhanced professional competence, perceived utility, fit with usual tasks and duties, fit with one’s own experience as a parent, fit with the mission of the organization, structure of the intervention, and an organizing conceptual model for the why and how of service delivery. Some case managers were motivated to use the EBP because they felt that their competence was enhanced by learning a new and evidencebased intervention. Case managers also evaluated the EBP in regard to its perceived utility. However, this perspective had to be garnered through actual experience with the EBP rather than what was taught in training. As one case manager stated, ‘‘I mean it’s required anyway. But when you see the results of how it works, it makes you want to continue to do it.’’ Another set of caseworker motives for using the EBP was its fit with their own experience of providing services. For instance, some caseworkers noted the fit between the EBP and their usual duties, tasks and responsibilities as demonstrated by the statement: ‘‘... I think I’ve always done it. It just hasn’t always had a name or paperwork to go with it...’’ This fits with a ‘‘common treatment elements’’ approach to providing services (Chorpita, Daleiden, & Weisz, 2005) but with a focus on current practices that might map onto elements of EBP models. Many caseworkers were motivated to implement the EBP because it was consistent with their own experience as a parent. As one participant observed, ‘‘... Well, if it worked for me, it can work for them.’’ The EBP was also perceived to be consistent with the mission of the organization. According to one of the caseworkers, ‘‘...our whole goal is to provide a safe, stable environment for the children and make sure they’re cared for and their needs are met. And those are the three main goals—home safety, health and the parent–child interaction—of the model...’’ This comment illustrates that notion of ‘‘innovation-values fit’’ (Klein & Sorra, 1996) in that implementation may be facilitated when the EBP or innovation fits the provider’s philosophy or approach to provision of services. Help for the Provider A third set of motives for using SC related to what it was able to provide for the individual caseworker. Case managers noted that the structured approach was helpful in organizing the way services were delivered. For some, the initial reluctance to implement the intervention was soon overcome by the realization that SC helped to provide structure to existing skills and services. For other caseworkers, the structured nature of the EBP model was seen Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 as helpful in organizing specific tasks and managing crises. According to one of the ongoing consultants, ‘‘...The nice thing has been having something to focus their attention on that has steps. So that’s been a good thing to kind of refocus for whatever crisis is going on and give them an actual plan of attack.’’ The structured intervention also provided a common language for case managers as well as a context for why certain activities in the EBP were useful and objectively driven, rather than being pejorative: ‘‘...so we’re all on the same page, you know, I’m not doing my own parenting per se technique with this family; I’m doing [the EBP] Parent Activity Training...’’ The EBP was viewed positively by caseworkers because of its flexibility or adaptability to some specific families and situations. The ability to adapt the delivery, even within the structured intervention was seen as an important positive determinant of implementation. For example, one case manager noted that ‘‘...I think it’s pretty flexible.... And I don’t do that word for word, because actually the model doesn’t require that...’’ and also stated ‘‘...and you can even adapt the role modeling to not make it quite so anal and offensive. I mean you really can...’’ This adaptability was seen as very important in tailoring the EBP to the needs and styles of the families that were being served. Experiences with Being Trained in EBP The process of training for the implementation of the EBP was another theme that emerged from the data. For example, the process of training and caseworker interaction with and perception of trainers was critical in shaping their responses. A sense of responsiveness was considered a positive aspect of the training experience. One of the case managers noted that the trainers ‘‘...have been really good in listening to us and relaxing a little bit—changing the format so it’s not so rigid...’’ However, there were concerns that the trainers did not have a deep understanding of what it is like to actually deliver services or that they were not real experts as was summed up in the following response: ‘‘I think [the trainer] lacks some social work skills...’’ Caseworkers were not passive recipients of the training and engaged in internal evaluation of the training quality, content and process. Earlier orientation and preparation might have improved openness to training. For example, one respondent stated that ‘‘...it would’ve probably done us a world of good if we had had the book [training manual], which we probably did and we didn’t know it. And could’ve read about the research model first. And read that maybe in advance, maybe like two weeks ahead of time to know what the premise was behind it all.’’ However, caseworker understanding of the rationale for implementation helped to promote buy-in for the model. While some ambivalence was expressed, the following example 415 illustrates the positive perception of the model: ‘‘...that it was research and that they thought that was one of the best models out there... It’s cutting edge...’’ Extent of Organizational Support for EBP Implementation The data collected here reflected acknowledgement of the importance of multi-level organizational support for implementation and its impact on case manager perceptions. For example, at the child welfare system level there was perceived support for the EBP model in the statement ‘‘...DHS wanted the [Parent Activities Training] done with this child...’’ At the agency level there was perceived leadership support for implementation. For example, ‘‘...Yeah, they’re real supportive and the agency that we work for, as far as they back us up and are there for us...’’ and ‘‘... Sometimes they’re way too supportive of [the EBP]...’’ At the supervisory level there was also support for the [EBP] model. For example, ‘‘...[My supervisor] is always keeping up. Where are you on it? Did you do the health/medical [module]? And she/he believes that it’s a good program...’’ Direct leader communication supporting the EBP was perceived as important: ‘‘... [My supervisor] and I have discussed [Parent Activities Training] extensively, because there were, even after the training, some areas where I wasn’t quite positive on how to implement it. So we discussed some real world examples.’’ This was also reflected in examples of positive relationships with supervisors: ‘‘... I couldn’t have lucked into a better [supervisor] if I would’ve handpicked her. She’s very supportive of the [EBP] model...’’ Finally, support from the ongoing consultants was valued: ‘‘...yeah, I think [the consultants], all the people I work with believe in it. And I think that’s very important.’’ Thus, leadership support for the EBP at multiple system and organizational levels supported implementation. Impact of EBP on Process and Outcome of Services Perceptions of the utility of the EBP varied markedly. While some case managers stated that the model was positive (e.g., ‘‘something solid, something that was researched’’), others felt that the more structured EBP hindered the ability of case managers to get their work done in an efficient way. For example, it was reported that ‘‘...[the EBP] is inefficient because you’re so busy trying to do it [the trainer’s] way. Because [my ongoing consultant] goes off and reports [on me] and tells people. She/he writes e-mails to the big hot dogs at [my agency]. And you don’t want that because you want to keep your job.’’ This sense of increased oversight and reduced job autonomy was seen as a negative aspect of EBP implementation. 123 416 Another important theme was how well the EBP worked with the service population. In regard to outcomes, there were perceptions that the outcomes were indeed positive: ‘‘...Yeah, it’s working miracles for my families...’’ But perceptions were circumspect and there was an understanding that the EBP may fit the needs of some families better than others. For example, it was reported by one case manager that ‘‘...with some families it works well. With some families it’s actually less than great...’’ This suggests ambivalence about the applicability of the EBP for the entire service population. This is a common issue when EBPs are moved from academic settings to real world service systems where clients may have a more complex range of characteristics, problems, and issues than intervention development samples. Discussion As identified above, provider perspectives on implementation represent six primary factors that fall into two broader themes of attitudes toward or assessments of the EBP itself and experiences with learning and delivering the EBP. The implementation process was characterized by the notion of reciprocal adaptation, that is, the perceived need for the EBP to be adapted and the need for providers to adapt their perceptions and behaviors to accommodate the EBP. The EBP may need to be adapted to the context and service populations while providers must learn and adapt to the model. While there are adherents to the notion that no adaptation of an EBP should be permitted, others suggest that adaptation will and must happen to fit the EBP to the local context or to fit the context to the EBP (Schoenwald & Hoagwood, 2001). We propose that varying degrees of both these perspectives characterize actual implementation in real world service settings. The results of this study suggest that the best laid plans are but a part of the process of implementation. From this perspective, implementation is viewed as an adaptive endeavor. It is unrealistic to assume that implementation is a simple process, that one can identify all of the salient concerns, be completely prepared, and then implement effectively without adjustments. It is becoming increasingly clear that being prepared to implement EBP means being prepared to evaluate, adjust, and adapt in a continuing process that includes give and take between intervention developers, service systems, organizations, providers, and consumers. Attitudes Toward or Assessments of EBP We found two broad dimensions to attitudes or assessments of EBP. The first is attitudes toward EBPs in general. In 123 Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 this study, we found providers to be generally open to EBPs or using new or innovative practices (Aarons, 2004). It is also important to understand both individual provider characteristics (e.g., personality, training, experience) as well as the organizational context (e.g., supportiveness, culture, climate) (Aarons, 2005). In regard to assessments of the EBP, providers were likely to be concerned with acceptability and fit with the needs and preferences of the family. This concern brings to mind the Institute of Medicine’s (2001) definition of evidence-based practice that includes a balance of the best research evidence, clinical expertise, and consumer choice and preference. Rigid implementation may not honor the spirit of this definition (Melnyk & Fineout-Overholt, 2006). The notion of the fit of an innovation with the implementation context and processes is supported in the broader organizational literature on implementation of innovation where an innovation is congruent with the approach, values and methods of the adopter, implementation climate will be enhanced along with implementation effectiveness (Klein & Sorra, 1996). Experiences with EBP Experiences with EBP involve a range of concerns including learning about the EBP, being trained in use of the EBP, perceived organizational and leadership support—which sets the stage for implementation, and the impact of the EBP on the process and outcome of case management. Organizational support for implementation can be found at multiple levels and is predictive of implementation effectiveness (Klein, Conn, & Sorra, 2001). If the EBP enhances the process of engaging clients and working with families, it is likely to be viewed more positively. In addition, if the EBP is seen as being instrumental in attaining outcomes that are valued by case managers (not just those of researchers or intervention developers) there is likely to be greater acceptance and more effective implementation of the EBP. For example, it was noted that court mandates increase client motivation, engagement, and compliance in the EBP thus supporting implementation. Although our understanding of EBP barriers and facilitators is limited by its focus on the perspective of case managers on the front lines of service delivery in the childwelfare system, we were able to examine a perspective that is often not assessed at an in-depth level. In addition, there are a number of lessons to be learned from this study that would be difficult to capture by more quantitative or fidelity measures. Service providers have a complex and difficult job. A myriad of work responsibilities occur in concert with the implementation of an EBP. Productivity and paperwork requirements are often at odds with service provision. In addition, it is rare that a clinician or case Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 manger will deliver only one specific intervention. The complexity inherent in a real world service population generally does not allow for that. The results presented above reflect struggles to provide services to a complex service population. Service providers are also diverse in regard to age, gender, race/ethnicity, education and training, and experience. In addition to the EBP being implemented, the professionals who provide services have opinions, likes, dislikes, and preferences. Implementation does not occur on a tabula rasa. We also found that adaptation of an EBP does not refer just to the desire or need to adapt an EBP to the local context. Rather, adaptation can also take place in the implementation context, and at the personal and interpersonal levels. At the contextual level, adaptation means that the organizational context must be altered to accommodate the EBP. This represents changes in daily routines, materials used, and productivity requirements. At the personal level, adaptation is how the service provider adapts to the use of the EBP. For example, ongoing consultants who monitor and advise case-mangers were initially seen as intrusive but later seen as coaches, resources, and aids in providing services. This might be accomplished through cognitive mechanisms of accommodation or assimilation (Phillips, 1969). If neither accommodation nor assimilation can be managed—then coping strategies must be used in order to facilitate continued job stability in the work setting. However, if none of these can be successfully employed—then it is likely that turnover intentions will increase and work performance will suffer. Indeed, it was noted in this study that increased structure and oversight reduced perceived job autonomy. At the interpersonal level, adaptation means that case managers must adapt to new or changed interpersonal relationships. For example, many EBPs have fidelity measures that may be completed by providers, supervisors, clients, or by observers. In the present study, fidelity is assessed by ‘‘ongoing consultants’’ who observe case managers in the field working with clients. This is an example of a new interpersonal relationship. Case managers also now have a more structured way of interacting with clients and this represents a changed interpersonal relationship. Relationships with coworkers and supervisors may also change. For example if a case manager does not embrace a new model but this is at odds with a supervisor or co-workers, the interpersonal dynamics of their work life will change. All of this might be brought about by the implementation of the new service model. It has been demonstrated, however, that organizational change itself can lead to increased staff turnover (Baron, Hannan, & Burton, 2001) and this further complicates the implementation process. One approach to implementation may be to take a continuous quality improvement perspective (Bishop & 417 Dougherty, 2005; Uretsky & Wang, 2006). This perspective involves a cycle of planning, action, assessment, and revision of plans and processes. In this way, the implementation process is a complex adaptive system (Bro & Kragstrup, 2003; Barriere, Anson, Ording, & Rogers, 2002; Jankowicz, 2000). This perspective views implementation as an active process rather than a static outcome. Such an approach values the experiences and input of all involved stakeholders in order to make services more relevant, efficient, and effective. While empirical data about factors that facilitate or impede EBP implementation efforts in human service settings is beginning to accumulate (Henggeler, Lee, & Burns, 2002; Morgenstern, 2000), it is clear that publishing evidence in the form of empirical studies, publishing guidelines based on evidence, or educating providers about the evidence, are strategies that fail to move the majority of providers beyond the threshold required to implement new practices with fidelity (Backer, Liberman, & Kuehnel, 1986; Burns, 2003; Kroenke, Taylor-Vaisey, Dietrich, & Oxman, 2000; Torrey et al., 2001). Systematic efforts, such as those described in this study, work to address the need for service change and the flexibility needed to effectively implement EBPs. Though some studies have found that providers desire more training (Lehman, Greener, & Simpson, 2002), it is unlikely that didactic training sessions alone will have a lasting impact on services and lead to implementation with fidelity. Simpson (2002) notes that lack of funds for attending conferences is seen as a major barrier to effective change in practice. However, the substantial organizational research on transfer of training suggests that more longterm and comprehensive strategies are needed in order to promote practice change (Cheng & Ho, 2001; Clarke, 2002; Dansereau & Dees, 2002). Insufficient resources have been seen as a main barrier with peer support, quality of supervision, and team meetings as facilitating or limiting implementation (Kavanagh et al., 2003; Milne, Dudley, Repper, & Milne, 2001). Studies in the physical health care sector have shown that providers are often unable or unwilling to implement EBPs or practice guidelines (Grol, 2001). Haynes and Haines (1998) suggest the use of multiple strategies including abstracting services, evidencebased clinical guidelines, incentives for better care systems, and increasing the effectiveness of quality improvement programs. Dixon and colleagues (Dixon et al., 2001) address dissemination issues at the policy level (focus on outcomes, cost-effectiveness, consumer satisfaction), clinician and program level (buy-in, work schedules, professional discipline, leadership, high caseloads, training resources, reimbursements, attitudes, knowledge), and consumer level (transportation, time, energy). Carpinello and colleagues (Carpinello, Rosenberg, Stone, Schwager, 123 418 & Felton, 2002) suggest that a communicable vision, implementing regulations regarding treatment, using opinion leaders, developing ‘‘centers for excellence,’’ fiscal and regulatory changes, and performance based outcomes are needed. However, research on the impact of such factors is currently minimal and results may be equivocal. For example, only meager evidence has been found for the effect of using local opinion leaders for change in practice (Thomson O’Brien et al., 2003). However, leadership is associated with provider attitudes toward adopting EBPs in mental health service settings (Aarons, 2006). Comprehensive theories of EBP implementation should take into account the exigencies of moving efficacious interventions into real world service settings. This study provides guidance on the perspectives of service providers that highlights attitudes towards and experiences with EBP as critical elements in the implementation process. Taken along with findings from other studies, this represents a new direction in the study of implementation of EBPs. There is a systemic learning process that takes place during implementation. Consideration of multiple levels including the system, organization, provider, and consumer (Ferlie & Shortell, 2001) is needed to improve the process and outcomes of EBP implementation. Acknowledgements The authors thank Dena Plemmons, Ph.D. for conducting field interviews and Tamiko Wong, B.A. for work on data coding. We also thank the participant case managers and trainers for their time and perspectives on implementation. This project was supported by NIH grants: R01MH072961 and R24MH067377. References Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. Aarons, G. A. (2005). Measuring provider attitudes toward evidencebased practice: Organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14, 255–271. Aarons, G. A. (2006). Transformational and transactional leadership: Association with attitudes toward evidence-based practice. Psychiatric Services, 57(8), 1162–1169. Aarons, G. A. & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidencebased practice. Psychological Services, 3(1), 61–72. Backer, T. E., Liberman, R. P., & Kuehnel, T. G. (1986). Dissemination and adoption of innovative psychosocial interventions. Journal of Consulting & Clinical Psychology. Special Issue: Psychotherapy Research, 54(1), 111–118. Baron, J. N., Hannan, M. T., & Burton, M. D. (2001). Labor pains: Change in organizational models and employee turnover in young, high-tech firms. American Journal of Sociology, 106(4), 960–1012. Barriere, M. T., Anson, B. R., Ording, R. S., & Rogers, E. (2002). Culture transformation in a health care organization: A process 123 Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 for building adaptive capabilities through leadership development. Consulting Psychology Journal: Practice & Research, 54(2), 116–130. Bickman, L., Heflinger, C. A., Lambert, E. W., & Summerfelt, W. T. (1996). The Fort Bragg managed care experiment: Short term impact on psychopathology. Journal of Child & Family Studies, 5(2), 137–160. Bishop, A., & Dougherty, R. (2005). Implementing CQI at the healthcare provider level. Behavioral Healthcare Tomorrow, 14(1), 38–39. Bro, F., & Kragstrup, J. (2003). From black box to complex adaptive system. Scandinavian Journal of Primary Health Care, 21(1), 1. Burns, B. J. (2003). Children and evidence-based practice. Psychiatric Clinics of North America, 26(4), 955–970. Burns, B. J., Hoagwood, K., & Mrazek, P. J. (1999). Effective treatment for mental disorders in children and adolescents. Clinical Child & Family Psychology Review, 2(4), 199–254. Carpinello, S. E., Rosenberg, L., Stone, J., Schwager, M., & Felton, C. J. (2002). New York state’s campaign to implement evidencebased practices for people with serious mental disorders. Psychiatric Services, 53, 153–155. Cheng, E. W. L., & Ho, D. C. K. (2001). A review of transfer of training studies in the past decade. Personnel Review, 30(1–2), 102–118. Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: a distillation and matching model. Mental Health Services Research, 7(1), 5–20. Clarke, N. (2002). Job/work environment factors influencing training transfer within a human service agency: Some indicative support for Baldwin and Ford’s transfer climate construct. International Journal of Training & Development, 6(3), 146–162. Dansereau, D. F., & Dees, S. M. (2002). Mapping training: The transfer of a cognitive technology for improving counseling. Journal of Substance Abuse Treatment, 22(4), 219–230. Dixon, L., McFarlane, W. R., Lefley, H., Lucksted, A., Cohen, M., Falloon, I., et al. (2001). Evidence-based practices for services to families of people with psychiatric disabilities. Psychiatric Services, 52(7), 903–910. Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5(1), 47–52. Ferlie, E. B., & Shortell, S. M. (2001). Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q, 79(2), 281–315. Filene, J. H., Lutzker, J. R., Hecht, D. B., & Silovsky, J. F. (2005). Project SafeCare: Replicating an ecobehavioral model of child maltreatment prevention. In K. Kendall-Tackett & S. Giacomoni (Eds.), Victimization of children and youth: Patterns of abuse, response strategies. Kingston, NJ: Civic Research Institute, Incorporation. Fraser, D. (2000). QSR NVivo NUD*IST Vivo reference guide. Melbourne: QSR International. Garland, A. F., Kruse, M., & Aarons, G. A. (2003). Clinicians and outcome measurement: What’s the use? Journal of Behavioral Health Services and Research, 30(4), 393–405. Gershater-Molko, R. M., Lutzker, J. R., & Wesch, D. (2003). Project Safecare: Improving Health, Safety, and Parenting Skills in Families Reported for, and at-Risk for Child Maltreatment. Journal of Family Violence, 18(6), 377–386. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter. Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of health promotion research to practice? Adm Policy Ment Health & Ment Health Serv Res (2007) 34:411–419 Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93 (8), 1261–1267. Glisson, C. (1992). Structure and technology in human service organizations. In Y. E. Hasenfeld (Eds.), Human services as complex organizations (pp. 184–202). Thousand Oaks, CA: Sage Publications, Inc. Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253. Grol, R. (2001). Successes and failures in the implementation of evidence-based guidelines for clinical practice. Medical Care, 39(8), 46–54. Haynes, B., & Haines, A. (1998). Barriers and bridges to evidence based clinical practice. Biomedical Journal, 317, 273–276. Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after the innovation is identified? Clinical Psychology: Science and Practice, 9(2), 191–194. Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and -dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1(3), 171–184. Hoagwood, K., Burns, B. J., Kiser, L., Ringeisen, H., & Schoenwald, S. K. (2001). Evidence-Based Practice in Child and Adolescent Mental Health Services. Psychiatric Services, 52(9), 1179–1189. Hurlburt, M., & Knapp, P. (2003). The new consumers of evidencebased practices: Reflections of providers and families. Data Matters, Special Issue #6, 21–23. Jankowicz, D. (2000). From ‘‘learning organization’’ to ‘‘adaptive organization’’. Management Learning, 31(4), 471–490. Jensen, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy of Child and Adolescent Psychiatry, 42(5), 527–530. Kavanagh, D. J., Spence, S. H., Strong, J., Wilson, J., Sturk, H., & Crow, N. (2003). Supervision practices in allied mental health: Relationships of supervision characteristics to perceived impact and job satisfaction. Mental Health Services Research, 5(4), 187–195. Klein, K. J., Conn, A. B., & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86(5), 811–824. Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055– 1080. Kroenke, K., Taylor-Vaisey, A., Dietrich, A. J., & Oxman, T. E. (2000). Interventions to improve provider diagnosis and treat- 419 ment of mental disorders in primary care: A critical review of the literature. Psychosomatics: Journal of Consultation Liasion Psychiatry, 41(1), 39–52. Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22(4), 197–209. Melnyk, B. M., & Fineout-Overholt, E. (2006). Consumer preferences and values as an integral key to evidence-based practice. Nursing Administration Q, 30(2), 123–127. Milne, D., Dudley, M., Repper, D., & Milne, J. (2001). Managers’ perceived contribution to the transfer of psychosocial interventions training. Psychiatric Rehabilitation Skills, 5(3), 387–402. Morgenstern, J. (2000). Effective technology transfer in alcoholism treatment. Substance Use & Misuse, 35(12–14), 1659–1678. National Institute of Mental Health. (1999). Bethesda, MD: National Institutes of Health. Phillips, Jr. J. L. (1969). The origins of intellect Piaget’s theory. W.H. Freeman and Company. Rotheram-Borus, M. J., & Duan, N. (2003). Next generation of preventive interventions. Journal of the American Academy of Child & Adolescent Psychiatry, 42(5), 518–526. Schoenwald, S. K., & Hoagwood, K. (2001). Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services, 52(9), 1190–1197. Simpson, D. D. (2002). A conceptual framework for transferring research to practice. Journal of Substance Abuse Treatment, 22(4), 171–182. Thomson O’Brien, M. A., Oxman, A. D., Haynes, R. B., Davis, D. A., Freemantle, N., & Harvey, E. L. (2003). Local opinion leaders: Effects on professional practice and health care outcomes. In The Cochrane Library, (Chap. Issue 1). Oxford: Update Software. Torrey, W. C., Drake, R. E., Dixon, L., Burns, B. J., Flynn, L., Rush, A., et al. (2001). Implementing evidence-based practices for persons with severe mental illnesses. Psychiatric Services, 52(1), 45–50. Uretsky, B. F., & Wang, F. W. (2006). Implementation and application of a continuous quality improvement (CQI) program for the cardiac catheterization laboratory: One institution’s 10year experience. Catheterization and Cardiovascular Interventions, 68(4), 586–595. Willms, D. G., Best, A. J., Taylor, D. W., Gilbert, J. R., Wilson, D. M. C., Lindsay, E. A., et al. (1992). A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly, 4(4), 391–409. 123