Evaluating Domestic Violence Programs (1997)

Evaluating Domestic Violence Programs Evaluating Domestic Violence Programs Domestic Abuse Project 204 W. Franklin Avenue Minneapolis, Minnesota 55404 © 1997 by Domestic Abuse Project All rights reserved The Domestic Abuse Project (DAP) reserves the right to modify the contents of this manual at any time. DAP shall not be liable for errors contained herein. This document contains proprietary information which is protected by copyright. No part of this document, with the exception of the forms in the Sample Measures I and II sections, may be reproduced in any form without prior permission from DAP. Editing and page design by Carol J. Frick Cover art/design by Ann L. Kranz Workshop—Evaluating Domestic Violence Programs is also offered as a oneday workshop with Dr. Jeffrey Edleson. Through discussion and lecture, participants build their understanding of how to evaluate their specific programs, and they gain valuable experience through group activities. For more information about this highly rated workshop, contact the Domestic Abuse Project’s general office using the numbers below. Publications—To order additional copies of this manual, or to receive information about other DAP manuals, a copy of the Training & Research Update, or a list of available research articles and reports, please contact the Domestic Abuse Project at: ii Telephone 612-874-7063 FAX 612-874-8445 E-mail dap@mndap.org Contents About the Domestic Abuse Project (DAP) Profile of Jeffrey L. Edleson, PhD vii This manual is for you if . . . viii About this manual ix Suggestions for using this manual x vi 1 Understanding the function of program evaluation: Why do evaluation? 2 Warm-up exercise—Reflecting on your experiences with evaluation 3 2 Making the decision: Should you evaluate your program? 6 Good reasons to do evaluation 6 When not to do evaluation 8 Exercise—Examining reasons for and against evaluating your domestic violence program 10 3 Laying the groundwork for evaluation: Is your program built on useful goals and objectives? 12 Exercise—Understanding your program’s current goals 13 How goals and objectives fit into the big picture 15 Refining your goal statements and outcome objectives as preparation for evaluation 16 Characteristics of useful goals and objectives 16 Program logic 20 Exercise—Define four goals and outcomes for your program 21 4 Assessing the evaluation environment: Does your organization have the ability to evaluate its program? 24 Evaluation basics—a quick tour 24 What it all means for your organization 25 Try a creative approach to meeting resource needs 25 The evaluator’s relationship to the organization and program can affect evaluation 26 iii 5 Measures and procedures: What are the right tools for the job? 28 Where will the data come from? 28 How will you collect the data? 28 When will you collect the data? 30 What are you measuring with these tools? 30 Exercise—Choosing and implementing evaluation measures Which measure is best? 34 Exercise—Evaluation design alternatives 35 Practical considerations for evaluation design 37 Ethical considerations for evaluation design 38 Learning to change focus from activity to impact 38 How data is collected at the Domestic Abuse Project 39 32 6 Modeling evaluation: About the DAP sample measures 42 Using the Therapy Unit follow-up forms 42 Using the Intervention Satisfaction Survey 43 Using evaluation for systems advocacy 44 7 Analyzing, presenting, and using results: What did you find—and who cares? 46 What can positive results do for you? 46 What if study results show that your program fell short of its goals? How do you judge success? 47 Present your results and conclusions for maximum impact 49 More ways to present data 52 Preparing your interpretation of results for your audience 53 Disseminating results—a DAP example 54 Exercise—Putting your results to use 60 8 Putting it all together: Guided exercise in developing an evaluation plan iv 6 46 9 Meeting future information needs: Do you need a computerized database? 72 What is a computerized database? 73 How complicated or detailed is your information? 74 Do you need a “flat” or a “relational” database? 75 Should you create your own database? 75 What to look for in a database management product 76 Should you hire a professional database developer? 77 What information should you include in your database? 78 DAP’s legal advocacy database 78 Glossary 85 Appendix A—Sample measures I Therapy Unit follow-up forms 87 Release form to contact partner 89 Release of information 91 Women’s Participation Agreement 93 Program Goals for Women 95 Instructions for Therapy Unit follow-up 97 Completion summary form 109 Abuser intake form: Part I 111 Abuser intake form: Part II 115 Closing summary form 125 Abuser follow-up interview schedule 129 Individual summary sheet: Abuser 137 Quarterly summary: Abuser 139 Victim/survivor intake: Part I 141 Victim/survivor intake: Part II 145 Closing summary form: Victim/survivor 155 Individual Summary Sheet: Victim/survivor 159 Victim/survivor follow-up interview schedule 161 Children’s Program evaluation follow-up questionnaire Quarterly summary: Victim/survivor 169 Appendix B—Sample measures II Intervention satisfaction survey 173 Intervention satisfaction survey for DAP Selected reading 167 171 175 183 v About the Domestic Abuse Project (DAP) The Domestic Abuse Project was founded in 1979 as the response to requests from battered women who needed services in addition to shelter. Since its inception, DAP has grown in size and staff. With over 20 staff, Dap maintains four main program components: ❿ Legal and systems advocacy—DAP provides legal advocacy services for victims of domestic assault immediately after an arrest and throughout the court process. To build a coordinated community response, DAP staff train 911 operators, police officers, judges, prosecutors, and probation officers on issues of domestic violence. ❿ Research and evaluation—DAP staff conduct academic research on the effectiveness of different responses to domestic violence. As part of the evaluation process, follow-up with adult therapy clients takes place at six months after completion of group in order to measure program effectiveness and to ask parents about the effectiveness of the groups for children. Feedback is solicited from advocacy clients once their court case is closed. Twice yearly, the Training & Research Update is published. This report communicates research and evaluation findings to 8,000 colleagues worldwide in the domestic violence field. ❿ Therapy services—Batterers can attend approximately 14 weeks of twice-weekly, open-ended groups to develop control plans to end their abusive behavior. Victims/survivors of domestic abuse can attend 18 group sessions to develop protection plans and begin to heal from the effects of domestic violence and abuse. A 12-week adolescent group is available to boys 13 to 18 years old who are abusive to family members or in dating situations. Age specific children’s groups are held weekly for 10 weeks where children learn to protect themselves, break the secret of family violence, and strengthen self-esteem. ❿ Training and publications—DAP focuses its training efforts on those who have the ability to improve outcomes—the professionals who encounter domestic violence in their work. Local and national training is provided to criminal justice personnel, mental health professionals, health care workers, chemical abuse counselors, other domestic violence program staff, as well as many other social service agency personnel. DAP has published three manuals for group work in the areas of men who batter, women who are survivors of domestic violence, and children who witness domestic violence. In development are manuals for working with adolescents and for providing services to women who are abusive. vi Profile of Jeffrey L. Edleson, PhD Dr. Edleson has extensive experience both as a program evaluator and as a trainer. He has directed the Domestic Abuse Project’s Research and Evaluation Unit for more than 14 years. His studies have included evaluations of therapy groups for abusers, legal advocacy services for victims/ survivors, and programs for children who witness domestic violence. He provides training worldwide on domestic violence programming and research. Dr. Edleson is also a Professor at the University of Minnesota School of Social Work and Director of the Higher Education Center Against Violence and Abuse. He has published more than 60 articles and several books on domestic violence, groupwork, and program evaluation. He has provided technical assistance to domestic violence programs and research projects nationally and internationally, including Israel, Singapore, and Australia. Dr. Edleson was a member of the National Research Council’s Panel on Research on Violence Against Women and is currently a member of the Expert Panel of the National Resource Center on Domestic Violence, Child Protection, and Custody, a program of the National Council of Juvenile and Family Court Judges. He is a consultant to the U.S. Centers for Disease Control and Prevention’s Family and Intimate Violence Prevention Subcommittee. Dr. Edleson is an Associate Editor of the new Sage Publications, Inc., journal, Violence Against Women, and has served on the editorial boards of the Journal of Interpersonal Violence, Journal of Family Violence, Aggression and Violent Behavior, and Violence UpDate. His books include Working with Children and Adolescents in Groups, co-authored with Sheldon D. Rose (1987, Jossey-Bass); Intervention for Men Who Batter: An Ecological Approach, co-authored with Richard M. Tolman (1992, Sage Publications); Ending the Cycle of Violence: Community Responses to Children of Battered Women, co-edited with Einat Peled and Peter G. Jaffe (1995, Sage Publications); Future Interventions with Battered Women and Their Families, co-edited with Zvi Eisikovits (1996, Sage Publications). A Phi Beta Kappa graduate of the University of California at Berkeley, Dr. Edleson received his Masters and PhD in Social Work from the University of Wisconsin at Madison. He is a Licensed Independent Clinical Social Worker in Minnesota and has practiced in elementary and secondary school settings, as well as several domestic violence agencies. vii This manual is for you if. . . . . .You are an administrator or staff member of a domestic violence program and you need answers to some specific questions about how your program is doing. You may have received questions from clients, judges, government or private funders and others about the effectiveness of your program. Maybe you have so far relied on intuitive or informal evaluations of your program’s outcomes, and now you are wondering whether a formal approach to evaluation would provide more accurate and useful answers. Have you already carried out a formal evaluation of your program, or are you now in the process of doing an evaluation? Then you may gain helpful insight by comparing your methods with those recommended in these pages. Maybe you have been approached by a researcher who wants to evaluate your program as part of his or her research project, and you want to know if this is a good idea. Or have you been informed from “higher up” that the continuation of your program will be decided on the basis of an outside evaluation? This manual is a valuable resource for you in all of the above situations. Practitioners in fields related to domestic violence, such as sexual assault and child abuse, will also find solid guidance in this manual. The specific examples throughout the manual focus on domestic abuse, but the basic principles of program evaluation are highly applicable across a broad range of social service programs. About this manual The purpose of this manual is to help you make informed decisions about doing evaluation, and to provide you with concrete ideas for evaluating a specific program or group of programs. In a clear and simple style, the issues, elements, and procedures of beginning evaluation are examined. You will learn how to develop goals and outcome objectives that will focus your program and facilitate productive evaluation. Benefits and drawbacks of program evaluation are laid out, along with guidelines for assessing your agency’s ability to conduct an evaluation. The basic evaluation process is mapped out in step-by-step fashion, complete with sample forms and questionnaires. Throughout this manual you are encouraged to focus on how your study results will be used. Finally, you will learn the most effective ways to present your findings to various audiences when your evaluation is finished. viii If you are being asked to cooperate with an outside evaluator, this manual will help you know what questions to ask about the proposed evaluation. It will give you a basis on which to decide, if you have a choice, whether to open your program to the evaluation. If you don’t have a choice, you will gain insights that will help you determine whether you are being fairly judged by an outside evaluation and how to gain some control over the process. Evaluating Domestic Violence Programs is based on 14 years of a unique collaboration between research and services. Whether your program is new or long established, you can gain a more intimate knowledge of it through the kind of evaluation explained in this manual. This knowledge can help you increase your effectiveness as an administrator. You can do it! Evaluation does involve a number of complex issues. The procedures of evaluation, on a practical level, do require you to be fairly well organized. But there is no need to feel intimidated by evaluation. Anyone who sits face-to-face with a client in direct service—as advocate, service worker, or in a similar capacity—is also, at some level, making judgments about success. Some kind of information is being collected: it might be verbal information, or it might be observations of the behavior of the survivor, the children, or the batterer. This is data collection. It may be intuitive, but it is also likely to be at least somewhat systematic. As each worker gains experience, she or he is probably doing some type of assessment: Is this man changing? Is this woman or are these children safer? Your goal as an evaluator is to become more explicit about this process. That’s what this manual is designed to help you do. ix Suggestions for using this manual ❿ Exercises—We encourage you to take time to do all of the exercises that you will find throughout these pages. By doing so, you will be starting on the path toward designing and implementing your own program evaluation. ❿ Training—Share what you are learning about evaluation with those who will be assisting in the process. Keep this manual handy throughout your evaluation planning and implementation so you can periodically review key topics such as goal development or uses of the various measurement tools. Also consider attending DAP’s Evaluating Domestic Violence Workshop for a personal approach to learning the information in this manual. ❿ Sample forms—You will find a wide variety of sample forms that have been developed and refined for evaluation of DAP programs. Even if your program is new, or you are new to evaluation, you can base your efforts on the Domestic Abuse Project’s many years of acquired experience in domestic abuse program evaluation. ❿ Gender-specific language—DAP works with people who are in heterosexual and same-sex relationships that involve violence. Most of our evaluation experience stems from our work with people in heterosexual relationships in which the violence is perpetrated by males against females in the majority of instances. In this manual, many references are made to men as the abusers and women as the victims/survivors, but we recognize that domestic violence also occurs in same-sex relationships. x 1 Understanding the FUNCTION of program evaluation Why do EVALUATION? Know your purpose. There are a multitude of reasons—good and bad, for and against—that go into a decision about conducting program evaluation. We’ll examine many of these reasons in the following pages. Being able to identify your own reasons—knowing your purpose—is the first part of good evaluation. Know your audience. An equally essential part of good evaluation is to focus on who will see and use your findings. This is your audience. If someone in particular is requesting or requiring evaluation, be sure you understand what the specific expectations are for information. The intended use of your results is your best guide to designing good evaluation. Make sure your results get used. Evaluation is worthwhile when it produces information that can lead to positive change. What specific impact do you want your evaluation to have? It won’t happen unless your findings reach the right people in a format they can use. You might be undertaking an evaluation in response to a specific funding requirement, but you can also increase the value of your efforts by seeking ways for those same results to serve others who have a stake in your program’s outcomes—think about clients, staff, the community, funders, policymakers, police, courts, the medical establishment, social services, and more. Specific plans for distributing and using results should underlie the entire evaluation process. 2 EVALUATING DOMESTIC VIOLENCE PROGRAMS Warm-up exercise Reflecting on your experiences with evaluation Take a few moments to write down some observations about any past experience you have had with program evaluation: ❿ What types of program evaluation have you already participated in? (Examples: Gathered verbal or written comments and suggestions from staff; distributed exit questionnaires to group members or shelter users; gathered data from court and police records; did six-month follow-up phone interviews with clients; cooperated with an outside evaluation conducted for research purposes.) ❿ What did you find useful about those evaluations? ❿ What was not helpful? ❿ Who used the findings? How were they used ? UNDERSTANDING THE FUNCTION OF PROGRAM EVALUATION 3 4 EVALUATING DOMESTIC VIOLENCE PROGRAMS 2 Making the DECISION Should you EVALUATE your program? You may be feeling that you don’t have a choice about whether to do evaluation. Private and public funders are placing higher expectations on domestic violence programs to evaluate and report on their effectiveness. As federal support for domestic violence programming increased under the Violence Against Women Act, so has pressure to evaluate those programs. Program staff also face a need to gather information for internal use. Becoming informed about the evaluation process will help you in planning when and how to best carry out evaluation. Some reasons for doing or postponing evaluation are briefly outlined here. Many of the issues they introduce are discussed in more depth in the rest of the manual. Good reasons to do evaluation Evaluating your domestic violence program can help your organization in many ways: ❿ Evaluation can help you meet external demands for information. If the results of your evaluation support your program or provide guidance for change, they may be useful when there is a funding question or requirement, accreditation requirement, or information request. ❿ Evaluation can help you answer the question: Is your program successful? This question is likely to come from many sources. Clients want to know if your program “works”—if it’s worth their time and effort and hope. Funders are no longer satisfied just to be told how many people you served last year. When deciding whether to give you money, they want to know if your program is “making a difference.” Judges, police, the medical community, social services—all have an interest in the effectiveness of your program. The public longs to hear of programs that work, and has little mercy for those that appear not to work. 6 EVALUATING DOMESTIC VIOLENCE PROGRAMS It might seem obvious to say that a program is successful if it is achieving the goals that have been set for it. Evaluation can tell you if goals are being met. But be careful! Dubious and ill-conceived goals are in abundance—make sure yours are not among them. Take time to examine and clarify goals before setting out to evaluate programs. Try to get a consensus on goals among all who have a stake in your program: clients, funders, managers, staff, board members, community leaders. We will look at good and bad goal setting in Chapter 3. ❿ The results of your study may provide valuable information that will help you develop or improve your program. You and your staff need to know how effective your program is in order to determine if changes are needed. Monitoring the implementation and outcomes of your program in order to improve delivery is the best use of evaluation. A well-designed evaluation can be useful for many years in helping you to continually refine and shape programs. ❿ You might discover unintended side effects—positive ones to build upon, or negative effects that you can work to eliminate. Unintended effects are difficult to find because you don’t know what you’re looking for until you stumble on it. But once identified, they can be measured through qualitative methods such as interviews. Example of an unintended side effect—When evaluating our children’s program at DAP, we learned that many of the mothers were bringing their children to our children’s group because the mothers felt ready to help the children heal by engaging them in discussion about the witnessed violence. An unintended obstacle for these mothers was our standard procedure at all first group meetings of explaining the confidentiality of group discussions. During interviews with the mothers, it became apparent that many of the children took this to mean they should not tell their mothers about the feelings they talked about in group. Now we make it clear that it is okay for the children to discuss what they felt or talked about in group, but they should not talk about other group members. ❿ Evaluation can provide program accountability. This is a significant and multifaceted reason for doing evaluation. It is important to be accountable to ourselves and our programs—to know if we are making a difference. It is important to be accountable to the various layers of community in which we operate, including meeting government’s requirements for accountability. Most important, we must be accountable MAKING THE DECISION 7 for victim safety. Evaluating our programs’ outcomes and, equally as important, making the results known to the people who have a stake in them, helps us be accountable for victim safety. Certainly victim/survivors, both women and children, are among our most important audiences who need to know the results of our studies. Putting accountability for victim safety into practice—At DAP, research on batterers’ programs is a high priority because it is apparent from Edward Gondolf ’s research on women leaving shelters that women are more likely to go back to their batterer when he has enrolled in a batterers’ treatment program. The women feel—and some programs promote the idea—that the problem is under control once he is enrolled in the batterers’ treatment program. But our studies do not offer such a clear picture of the success of batterers’ programs. Consequently, we are conservative in what we tell both the men and the women about the likelihood that the man will stop being violent, especially keeping in mind the high dropout rate that is typical of batterers’ programs. When not to do evaluation Not all reasons are good reasons to do evaluation. And even when you have sound reasons, the timing may be inappropriate. Following are indicators that this is not a good time for you to subject your program to evaluation: ❿ It’s tempting to use evaluation as an excuse to postpone decisions or to shift decision making to the evaluator. But this is not an appropriate reason to conduct evaluation. ❿ If you don’t have specific questions about your program at this time, evaluation will serve little purpose. To design a useful evaluation, you should have a clear idea of what kind of information you want from it. Know in advance who will be using the information and for what purposes. ❿ If fueling public relations efforts is your primary reason for evaluation, this is probably not a justifiable use of time and resources. Refining programming and being accountable to funders and consumers are appropriate goals for conducting evaluation. Public relations gains may be a valuable additional product of evaluation, but when public relations is the primary goal there can be a conflict of interest between that and the honest reporting of results. 8 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Consider the political context in which your program operates. Evaluation could lead to an inopportune increase in public awareness of your program. Results of your study might provide ammunition to those wishing to attack your program. Is “protecting turf” an issue, or gaining a competitive edge? These are all reasons to rethink plans to evaluate. ❿ Does your program lack clearly articulated goals? A welldesigned evaluation is an excellent tool for measuring a program against its goals. It can produce information to serve as guidance for planning and implementing improvements. This opportunity may be lost if the program lacks well-defined goals. If staff, clients, funders and others with an interest are not in agreement about the program’s goals, evaluation results will be of questionable use. Suggestions for clarifying goals and outcome objectives are covered in the next chapter. MAKING THE DECISION 9 Exercise Examining reasons for and against evaluating your domestic violence program ❿ Why do you want to evaluate your program at this time? Write down your three main reasons. 1 2 3 obstacles to evaluation 10 EVALUATING DOMESTIC VIOLENCE PROGRAMS ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ❿ Are there any indications that this is not a good time to evaluate your program? (If needed, review the “When not to do evaluation” section on page 8.) If you do foresee potential obstacles, write them down along with an idea for overcoming each. solutions 3 Laying the GROUNDWORK for evaluation Is your program built on useful GOALS and OBJECTIVES? You might wonder why, in a manual about evaluation, a lengthy discussion is devoted to program goals and outcome objectives. Goals and objectives help focus a program. The importance of having consensus on a set of clear and realistic program goals cannot be overemphasized. Programs with clear goals are the ones that will reap the most benefit from evaluation. But it must also be recognized that both goal development and evaluation are learn-as-you-go processes. You probably already have experience in operating according to program goals at some level. Maybe you have an “official” set of goals in print form. Or maybe your program operates according to assumptions that haven’t been put into words, but they nevertheless drive your daily activities. Whatever the case, it’s worthwhile examining existing goals to make sure everyone is clear about them, and to determine if they need to be modified. Begin by considering who had input in defining your existing goals: one person? a committee? each person defines the goals according to individual perspective? Then consider who should have a part in defining program goals, and how to go about engaging those people in the goalsetting process. Consider that everyone who has an interest in your program—from clients, staff, and community, to funders and policymakers— should have input in the goal-setting process. Consensus on goals strengthens programs. 12 EVALUATING DOMESTIC VIOLENCE PROGRAMS Exercise Understanding your program’s current goals ❿ In your own words, describe what you feel are the primary goals of the program(s) you administer: ❿ Talk with staff members, individually or as a group, about what they feel the goals of the program(s) are. Summarize their comments: LAYING THE GROUNDWORK FOR EVALUATION 13 ❿ Gather and review your agency’s written goals for your program(s). Summarize them here. Are the written goals reflected in the responses gathered in questions 1 and 2? ❿ Does everyone on your staff share a cohesive understanding of the program goals? Or does each person interpret and implement the goals differently? ❿ What are some main points that everyone seems to agree on? ❿ What are some differences of viewpoint, if any? 14 EVALUATING DOMESTIC VIOLENCE PROGRAMS How goals and objectives fit into the big picture social programs Social programs aim to solve social and individual problems. goals Goals are general statements that summarize the ultimate impact that the social program is supposed to have on client problems. The general nature of goals usually makes them difficult to measure or fully attain. Each goal is likely to have multiple objectives. Your program will probably have multiple goals, which should be defined before developing outcome objectives and measures. Your own work may very well be inspired by a personal goal of participating in social change, but for evaluation purposes, program goals should be more concrete. outcome objectives Outcome objectives should fit within the scope of stated goals. Outcome objectives are the immediate, practical effects that you want your services to have on the problem. Ask yourself how clients or communities will be different after completion of the program compared to when it began. What will have changed in people’s thinking, in their behavior, in their lives? Objectives are realistic and attainable in light of the services provided. Objectives are measurable and are stated in a way that includes a level of achievement (often in terms of increase/decrease by a specific number or percent) within a given time frame. LAYING THE GROUNDWORK FOR EVALUATION 15 Refining your goal statements and outcome objectives as preparation for evaluation Getting everyone to specify and clarify goals, and to come to general agreement on them, is an important and usually challenging task to work on before attempting to design a program evaluation. You might even encounter some fear and resentment. Some people fear that articulating a meaningful goal—and isn’t “ending violence” our true goal?—automatically consigns programs to failure because the task appears overwhelming. Success can be made achievable by declaring goals that specify the intended impact of your program on client problems, and then setting manageable and measurable objectives that will move you toward those goals. There might be some people who resent the questioning of goals and the idea of evaluation because they feel these activities imply a lack of respect for the work that has been done. It is the evaluator’s task to educate and involve everyone in the evaluation process, focusing on how results can be used to benefit programs, and thus clients and society. An added incentive to goal development is that it can position you to head off undesirable situations. People who lack knowledge about domestic violence may want to impose goals that are not realistic and may even be counterproductive to your work. Being smart about setting goals and doing evaluation can help you to protect your program and your clients. Don’t be discouraged if you have difficulty articulating your program’s goals to everyone’s satisfaction. Clarify them as best you can for now. Meanwhile, it is reasonable to move ahead with program evaluation with the awareness that you will likely find yourself refining goals and objectives over time as you take in new information and experience. Characteristics of useful goals and objectives Use and share these guidelines as you work toward clarifying and building consensus around goals and objectives. ❿ It is to be expected that your program will have multiple goals. This allows you to accommodate the range of activities of your programs, and the various interests of clients, staff, policymakers, and others. ❿ Express your goals in ordinary language. Not everyone with an interest in your program’s goals will be familiar with the jargon specific 16 EVALUATING DOMESTIC VIOLENCE PROGRAMS Characteristics of useful goals and objectives (continued) to your field. If you must use some language unique to your field, remember to include a definition of terms. ❿ It’s often useful to state goals in terms of comparisons. Are you comparing your target outcomes to outcomes in the same program a year ago? Are you comparing target outcomes to another group of clients, former clients, or people in the community? ❿ Program goals should be specific for those people you have contact with. Such contact will most likely be in trainings, in community events, or in the services you provide. The mission of the battered women’s movement is to create social change—that is, through our combined efforts we aim to eradicate oppression and violence—but a single program by itself should not be expected to accomplish the whole task. ❿ Examine the values reflected by your goals. Be sure you are satisfied with these implied values. ❿ Identify only important goals and objectives. Avoid measuring unimportant goals and objectives just because they are easy to measure. ❿ Goals should be specific enough to be meaningful. Goal statements are somewhat more abstract than objectives, but they should still be measurable at some level. EXAMPLE OF OVERLY GENERAL VS SPECIFIC GOAL (MEN’S PROGRAM) Poorly conceived goal: The Domestic Violence Program will have a positive effect on the community. (The above goal statement is too general; it can’t be measured.) Better: Abusers who complete the Domestic Violence Program will end the use of violence against their partners. (This goal statement is more specific. It includes the element of increase/decrease, though it lacks the concreteness of number or percentage. This goal can be further defined through a series of outcome objectives.) LAYING THE GROUNDWORK FOR EVALUATION 17 Characteristics of useful goals and objectives (continued) Goals for men’s programs—Some might question the practicality of the zero violence goal illustrated in the second part of the example on the previous page. At the Domestic Abuse Project in Minneapolis, we feel it is reasonable to have a goal statement that the men will end their use of violence. This would be over an extended period, not just during the length of the program. You should specify a practical follow-up period. Research by Frank Dunford on police response shows that 95 percent of the recidivism is likely to occur in the 18 months after intervention. Zero violence represents a level of change that is of practical significance in the lives of victim survivors. Some people might be inclined to praise a statistically significant change such as a decrease in use of violence from six times a week to three times a week, but the practical reality is that the woman is still being abused. That’s why we stress zero violence as a goal for each man in the program. Goals for women’s programs—It is probably not reasonable to set for your women’s program a goal that you will end violence in the women’s lives. Women don’t have control over what their abusers do. Some states are imposing on women’s shelter programs a goal that the women will be less exposed to violence as a result of the program, but this is an unfair expectation since it presumes that the woman can control her abuser’s behavior. EXAMPLE OF A REASONABLE GOAL FOR A WOMEN’S PROGRAM Goal: Women who complete DAP’s group program will have increased their ability to protect themselves and their children. REASONABLE OBJECTIVES THAT DEFINE THE ABOVE GOAL Objective: Ninety percent of the women receiving support group services will be able to name four or more safety resources available to them in their community. Objective: At six months after the end of support groups, 90 percent of the women interviewed will be able to describe at least three elements in their safety plans. Objective: Ninety percent of women identifying a need for safety will be able to identify at least one element of their safety plans that they have used. 18 EVALUATING DOMESTIC VIOLENCE PROGRAMS Characteristics of useful goals and objectives (continued) In the example on the previous page, note that these objectives aren’t limited to the woman just receiving information (activity); they also include what action she will take if needed (outcome). ❿ Focus on outcomes rather than activity. Keep asking yourself: What kind of impact does the program have on the client’s life or on our community? Often organizations will report that “we made 3,000 calls; did 800 intakes; offered this many groups to that many people...”—but this does not describe how the clients’ lives or the community have been changed. It is important to gather activity data, but the bottom line is outcomes: Is it helping women and children live more safely? EXAMPLE OF “ACTIVITY” VS “OUTCOME OBJECTIVES” Poor: Sixty men who batter will receive our agency’s services over the next 12 months. (Focus is on activity.) Better: Sixty men who batter, and who receive our agency’s group program over the next year, will be reported by their partners as nonviolent and not threatening at six months after group completion. (Focus is on outcome.) LAYING THE GROUNDWORK FOR EVALUATION 19 Program logic Well-designed programs are characterized by an inherent logic that begins with clear and realistic goals. Your goals are made more concrete in specific, measurable objectives that direct you to mount specific program activities. From these activities, you should be able to measure levels of program activity (process) and their effect on participants (outcome). Since your activities are inspired by your goals and objectives, what you measure should reflect back on your goals. The logic of a well-designed program GOALS OBJECTIVES ACTIVITIES MEASURES Is there consensus What are the Do program Are process and on clearly stated immediate practical activities reflect the outcomes measur- and realistic effects? implementation of able and likely to program goals? result from program program goals? activities? Do measures reflect program goals? Does your program fit this model? 20 EVALUATING DOMESTIC VIOLENCE PROGRAMS Exercise Define four goals and outcomes for your program Using the information about goal setting that has just been covered, list at least four goals and outcomes that your domestic violence program should meet. GOAL #1: Outcome objective for goal #1: GOAL #2: Outcome objective for goal #2: GOAL #3: Outcome objective for goal #3: GOAL #4: Outcome objective for goal #4: LAYING THE GROUNDWORK FOR EVALUATION 21 22 EVALUATING DOMESTIC VIOLENCE PROGRAMS 4 Assessing the evaluation ENVIRONMENT Does your organization have the ABILITY to evaluate its program? Your program goals are in place and you’re ready to jump into the procedural details of evaluation. Or are you? To determine if your organization is in a position to undertake evaluation without disrupting services, you need to be aware of the issues and activities involved. Evaluation basics—a quick tour The following six steps are a brief overview of the evaluation process. The first two are covered in Chapter 1 and Chapter 3. The rest are discussed in the chapters that follow. Keeping all of the steps in mind throughout the evaluation process will help keep you on track as you sort out issues and make decisions at each stage. 1 Identify purpose and audience. Earlier in this manual you had the opportunity to examine your reasons for wanting to do evaluation. You were asked to identify the various audiences who have an interest in your program, and to consider what kind of information they need. 2 Identify goals and outcome objectives. The program goals and outcome objectives that you clarified in Chapter 3 will suggest the nature of the questions you can expect evaluation to answer. 3 Identify your data source. Soon it will be time to consider where you will get the information that holds the answers to your questions. You will have to decide if the information will come from staff, children, survivors, batterers, participants in community education programs, or from agency documents, observation of services, talking to referral sources, or yet other sources. 4 Collect the data. Where you get your information will largely determine how you collect it. If your data source is children, consider what might be the most appropriate method to obtain information—telephone interviews? questionnaires? observations? Don’t forget the timing. Will you collect the information before, during, or after the program (how long after)? You must also decide who will collect and report the information. 24 EVALUATING DOMESTIC VIOLENCE PROGRAMS 5 Analyze the data. When the data has been collected from the designated sources, it will need to be summarized and then interpreted in a manner that will be useful to your various audiences. Evaluation is not over until results have been disseminated in a useful form. 6 Make use of the findings. Always keep in mind the ultimate purpose of evaluation: to produce information that can be used to bring about positive change. What it all means for your organization It quickly becomes apparent that there will have to be some personnel and material resources directed to your evaluation project. It’s obvious that plans have to be formulated and many decisions need to be made. Someone must assume responsibility for overseeing the evaluation. In short, the process has to be managed from beginning to end. Good management is an essential key to successful evaluation. Try a creative approach to meeting resource needs Increasingly, funders are requiring program evaluation but few are providing financial resources for it. Some are allocating a portion of funds for evaluation, which might seem to be taking away from programs, but it is important to recognize that some personnel and material resources are needed to do evaluation. Consider including it in funding requests. You might have to look to a variety of sources for personnel. Evaluation can be done quite economically if you approach it in an organized manner. To use the Domestic Abuse Project as an example, we’ve had a one-quarter time person conducting follow-up phone calling for our direct services program. We have also at times used volunteers—formerly battered women who wanted to do something for the program—to do phone followup. We have used students in university work-study programs and in Urban Corps programs in which salaries were federally subsidized. As the availability of this help has decreased, we have shifted to having interns and staff doing various parts of the evaluation. Administrative staff prepare names and assemble files; direct service staff (not people who work with the individual client, but others such as interns) do the phone calling and interviewing; finally, administrative staff summarize the collected data. Spreading the work among staff members helps to keep costs down. ASSESSING THE EVALUATION ENVIRONMENT 25 The evaluator’s relationship to the organization and program can affect evaluation The evaluator may be an insider (program staff, specialist staff) or an outsider (consultant, monitor, funder). Either role has potential for advantage and disadvantage. Some evaluators are able to combine the best of both. Insider—The inside evaluator probably brings in-depth understanding of the program and enjoys the confidence of the administrative staff. His or her power to make necessary decisions might be limited by position. This evaluator may be concerned about a perceived or real job threat, or may approach evaluation with a bias toward the program that clouds objectivity. On the other hand, an insider may be in a better position to use study results to advocate for program change. Outsider—This evaluator may have been invited to conduct a study at the request of the organization, or perhaps was imposed upon the program by an external requirement. Or he or she may have applied for access to the program in order to conduct the evaluation. The outside evaluator could be hampered by lack of staff cooperation or difficulty in navigating the organizational culture. But he or she may also bring the objectivity and autonomy necessary to produce accurate results. Collaborator—A mutually beneficial relationship may result when an outside researcher expresses interest in collaborating with an agency. Collaboration means that both sides will have input into the research, from designing evaluation to using results for program development or improvement, thus meeting the needs of both research and practice. A basis on which to decide whether to allow research access to programs might well be whether the outside researcher is willing to commit to a long-term relationship of collaboration. At the very least, the researcher should be expected to provide, in exchange for research access, information that may enable the program to improve its services. 26 EVALUATING DOMESTIC VIOLENCE PROGRAMS 5 MEASURES and procedures What are the right TOOLS for the job? Program evaluation is turning data into knowledge. Before you can have knowledge, you need data. The data used in evaluation is produced by systematically collecting evidence of (measuring) your program’s outcomes. Where will the data come from? ❿ People—Much of the data you need will be provided by adult and child clients, staff and other observers, perhaps referrals (from and to), people you have trained, etc. ❿ Services—Some data can be obtained through observed sessions or activity statistics. Be aware that activity statistics by themselves are of limited use. Focus on measuring effects on the problem. How will you collect the data? Choice of collection methods, called measures, will be influenced by the nature of the data: ❿ Quantitative data is structured information measured in numbers. An often-used measuring tool for collecting quantitative data is questionnaires (surveys), either standardized or custom-designed. Typically, questions are responded to by checking off a number within a given range to indicate frequency or degree. Quantitative data provides breadth; collected from many people and/or collected over time, it can yield useful information of a comparative nature. ❿ Qualitative data can be categorized as semistructured, unstructured or open-ended information. Interviews and observations are tools often used to produce information of a qualitative nature, which provides more depth than quantitative data. Interviews from many different people would typically be analyzed for common themes. One useful result might be the identification of an unintended outcome. 28 EVALUATING DOMESTIC VIOLENCE PROGRAMS Following is a variety of qualitative and quantitative measures: ❿ Documents—Archival data can be obtained from existing records and files of your agency or other agencies. Examples: police records, court records, government statistics. Don’t be surprised to discover that much data is missing, even from your own agency. Also, the usefulness or accuracy of the information might be compromised by changes in values, data collection methods, or other factors. These are likely to be just some of many less-than-ideal conditions under which evaluation is necessarily conducted. ❿ Observations and ratings—Observe outcomes and then use a rating system to assign a score of some type. Observation is often costly and time-consuming, but it can produce important information about child, family, or group interactions that might be difficult to acquire through other means. Use rating scales or develop self-anchored rating scales in which clients determine the rating scale on an individual basis. ❿ Standardized questionnaires—Brief questionnaires developed to measure specific problems, often used repeatedly to measure change over time. A good resource for brief standardized questionnaires is Evaluating Practice: Guidelines for the Accountable Professional by Bloom, Fischer and Orme (Allyn and Bacon, 1995). Many standardized questionnaires are easy and quick to use, and for that reason are sometimes overused. They should not be expected to provide in-depth understanding. Also, there are very few designed to be sensitive to domestic violence issues. Keep in mind that some questionnaires have been designed to be accurate for comparison to large groups, but may be less useful for individual use. ❿ Questionnaires—Your agency might choose to develop its own “in house” questionnaires that are then given or mailed to staff and/or clients, or completed through telephone interviews. Often a great deal of thought must go into the wording of questions. At DAP, we have spent as long as a year revising a questionnaire until we were confident that it was an accurate measure of target outcomes. ❿ Client logs—Use of diaries and similar recordings by clients to note events that preceded, occurred during, and followed incidents, along with related feelings and thoughts. May produce useful information that is beyond the reach of other measures, but clients may be inconsistent in making recordings or distort information in order to meet perceived expectations of program staff. Evaluating Practice by Bloom, MEASURES AND PROCEDURES 29 Fischer and Orme, cited above as a resource for standardized questionnaires, also provides helpful information on using client logs. ❿ Interviews—Using structured, semistructured, or unstructured interviews and taking notes or generating transcripts that can later be analyzed. In structured interviews, each participant is asked the same questions, using the same wording. Semistructured interviews cover a specific list of topics with each participant. Unstructured interviews can be as open-ended as “tell me about your experience.” The latter are useful for gathering information that is complex or emotional, but resulting data is affected by the communication skills of both interviewer and client. ❿ Focus groups—Bringing together people from different groups, after they have finished the program, to talk about what they liked and didn’t like. A focus group often yields information that other measures miss. During one focus group at DAP, men gathered for an open-ended discussion of the program they had just completed, and it came up that several of them had felt there wasn’t enough processing time during group meetings, so after each session they would go to a nearby restaurant to continue their discussion, and this continued for many months afterward. This is an example of the kind of information that surveys or even structured interviews may not reveal. When will you collect the data? Useful data is usually time-related. Times that often yield good data include: before intervention (pretest), at end of service (post-test), and at follow-up. Timing of follow-up, whether at 6, 12, or 18 months, for example, is related to the goals of evaluation and limited by practicality. What are you measuring with these tools? What is it that you want to know about your program? That is what you measure. Impact should be highest on your list. Figure out a way to measure the degree to which the program is having an effect on the larger problem. Also examine the longer term results of intervention. Effectiveness is also a high priority measure. You want to determine how well an objective is being accomplished. Relate actual results to expected results. 30 EVALUATING DOMESTIC VIOLENCE PROGRAMS In gathering information about people, we can make most use of four primary sources of information: behavior, feelings, opinion, knowledge. When designing measures, think of how to tap each of those sources. Information can be gathered in a variety ways, such as through in-depth interviews, observation of clients in their natural environment, or roleplaying activities. Pencil and paper methods such as questionnaires are often relied on because they are economical and easily administered on a group basis. Some examples of tapping the four sources of information: ❿ Behavior—Include questionnaire items that ask how frequently specific behaviors (such as name calling or shoving) occurred over a specific time period (such as in the last month). ❿ Feelings—Include one or more emotion-related items in a questionnaire, or use standardized questionnaires or inventories or selfanchored rating scales to assess emotional states such as depression. ❿ Opinion—Include questionnaire items asking client whether he agrees or disagrees with statements such as “Slapping your wife/partner is domestic violence.” ❿ Knowledge—Include questionnaire items such as ones that ask the client to list the safety resources that she is aware of that are available to her and her children. MEASURES AND PROCEDURES 31 Exercise Choosing and implementing evaluation measures For each of the four goals you listed in the exercise at the end of Chapter 3, choose one of the objectives you identified and list at least one measure and how it will be implemented. Goal #1 Identify objective: Measure: Implementation: Where will the data come from? When will you collect the data? Goal #2 Identify objective: Measure: Implementation: Where will the data come from? When will you collect the data? 32 EVALUATING DOMESTIC VIOLENCE PROGRAMS Goal #3 Identify objective: Measure: Implementation: Where will the data come from? When will you collect the data? Goal #4 Identify objective: Measure: Implementation: Where will the data come from? When will you collect the data? MEASURES AND PROCEDURES 33 Which measure is best? Give this consideration plenty of thought. Rather than looking for a single best measure, think about using multiple measures to gain different perspectives of your outcomes. Some combination of quantitative and qualitative measures might give you a more accurate and useful picture. There is rarely one best measure that will yield all of the information you need. The main thing is to stay focused on the question you want answered. Don’t try to make your question fit some particular method that you favor. Instead, seek the most useful measures for the job. Every measure has its limitations. Would it be sufficient just to look at police arrest reports when looking at self-reports by women is likely to reveal a very different picture? Combining perspectives might yield better information. The technical term for this approach is “triangulation.” Some programs collect the batterers’ reports of their violence at followup. At DAP, we feel our top-priority data is from the most recently identified woman partner of the man who has gone through the program. Studies at DAP and elsewhere show that men will report approximately 50 percent less violence against the woman than will the women in those same relationships during the same period. We take the women’s reports as the most realistic data available, better than the men’s reports. A limitation of this data is that it doesn’t ensure that we get the most recent victim, who could be someone other than the most recently identified partner. But we feel this data is probably the best we can do within practical constraints. 34 EVALUATING DOMESTIC VIOLENCE PROGRAMS Exercise Evaluation design alternatives This exercise contrasts qualitative and quantitative data. Read the sample measures on this and the following page. Consider how you might use both types of information in your evaluation. SAMPLE OF A QUALITATIVE MEASURE A quote from an evaluation interview with a nine-year-old girl: “You know, if I had kids like me, I’d probably send them to a group like this too. They’d learn to like it and want to go again, I hope. Well, maybe they wouldn’t. I don’t know. So they’d probably feel nervous about things like this too. So, ya know, I’d have experience at how they were like, ya know, they would have a better way—easier way to get it. So they know that other people have had the exact same experience. Well, maybe not the exact same, but almost same.” ❿ If you were evaluating a group program for child witnesses of domestic violence, what themes might you draw from the above quote? ❿ Do you find yourself having difficulty analyzing the interview? ■ In fact, this type of data is usually much harder to summarize than is a checklist with numbers. In qualitative data, a “theme” is some observation— such as an experience, problem, or feeling—that is evident across a significant number of the interviews. In this example, there seems to be some indication of positive change taking place. Follow-up would be needed to get more specific information about possible good things that have resulted from the program experience. Doing that across an entire group of children who received the service should help you draw some conclusions about the program and about both intended and unintended results. Now turn the page to look at a very different kind of data. MEASURES AND PROCEDURES 35 The following sample measure resembles some that are commonly used in evaluating children’s programs. Read each of the six statements and circle a number as though you were a parent or teacher rating a particular child. SAMPLE OF A QUANTITATIVE MEASURE For each of the following statements, circle the 1 if it is rarely or never true of your child. Circle the 2 if the statement is sometimes true of your child. Circle the 3 if the statement is usually or always true of your child. 1 2 3 Has difficulty making friends of his/her own age. 1 2 3 Is a target for teasing by other children. 1 2 3 Spends free time alone. 1 2 3 Avoids physical games and sports. 1 2 3 Feels more comfortable playing with younger children. 1 2 3 Is quiet and withdrawn in a group of children. ❿ Can you draw a conclusion about the scores you circled? ❿ If you circled all 3s at intake, what change would you want to see at follow-up? ■ The kind of information that is produced by a questionnaire like this is something like a thermometer that the doctor uses to take your temperature. The thermometer will, over time, tell the doctor whether your temperature is going up or down, but it doesn’t offer a clue as to what’s causing the change. So the doctor interviews you, and maybe does some further tests, thus using a variety of measures to try to arrive at a diagnosis. Similarly, the questionnaire may provide a sense of shift over time across a group of children, but it won’t give you the depth that interviews do. This is the argument for using multiple measures to get the most accurate information. 36 EVALUATING DOMESTIC VIOLENCE PROGRAMS Practical considerations for evaluation design ❿ Cost—The cost of a measuring procedure, in terms of staff time and agency budget, usually has to be weighed along with other factors. ❿ Intrusiveness—From a research perspective, it might seem like a good idea, for example, to randomly assign 300 men to different program formats over a year’s time. But what happens when men in the long format begin questioning why other men are assigned to a short program, and when some women wonder why their partners receive fewer or less interesting-looking program materials than other men? Researchers call it “contamination of conditions.” For direct service workers, it’s a headache. Design the least intrusive evaluation that will produce useful information. ❿ Accuracy—Are the measures valid and reliable? That is, how well do they identify expected and unexpected changes and do they measure these in a stable way over time? Do clients and staff censor their answers to evaluators (reactivity)? ❿ Language—Consider how your use of language may influence responses. You may get a negative response if you ask, “Did you use your safety plan?” But you may get a list of things if you ask, “What did you do to help yourself and your children be safe?” You may get a denial despite every indicator pointing to the affirmative if you ask, “Are you being battered?” Many women simply don’t think of themselves as battered women even when they fit the criteria. Using concrete, jargonfree language will promote communication. ❿ Implementation—Easy-to-understand instructions should be provided for data collection. (For an example, see the “Instructions” portion toward the beginning of Sample Measures I in Appendix A, which begins on page 87.) ❿ Timing—The various audiences with an interest in your program are likely to be working on different timelines. Plan your evaluation so that results will be available at a useful time. Your study won’t have an impact if legislators, for example, receive the results after they had to make decisions that affect your program. MEASURES AND PROCEDURES 37 Ethical considerations for evaluation design ❿ Confidentiality—Be sure that your use of data in your evaluation does not violate client or agency confidentiality. ❿ Reporting—The “duty to warn” is particular to violence-related research. Have a protocol in place to handle mandatory reporting issues. Train your data collectors to know what to do. ❿ Misplaced values—The need for safety of the woman and her children must always be the priority of domestic violence programs. Measuring outcomes such as marital satisfaction and reunification imposes values that often overlook safety needs. ❿ Alternatives to no-service—A researcher from outside the domestic violence field might tell you that you must have a control group randomly assigned to no-service—otherwise how will you know if you make a difference? This approach may put your agency at risk ethically and legally and cannot be recommended. Consider instead a comparison study. Almost everyone has alternatives. The reality is that even “no service” clients are likely to find other helping mechanisms. A comparison study could be done of the various alternatives to your program that are available to clients. That way you will be able to provide a realistic evaluation of your services. Learning to change focus from activity to impact In a study we did at DAP in the late 1980s, we used court and police records to try to measure the effectiveness of advocacy programs. In three communities served by those early DAP advocacy programs, we found that arrests, prosecution, and guilty pleas had increased, and court ordered enrollment of abusers in abuser groups had increased. But the study didn’t tell us what kind of impact this had on violence. It didn’t provide any information about the effectiveness of the safety planning that was being done, such as what resources the women had that they didn’t have before. And it didn’t provide any insight into how the women viewed their experience with the criminal justice system. We could show that activity had changed significantly, but we didn’t have any data on the impact of that activity on the client group. Later, seeking more useful information, we did a study based on interviews with the women immediately after police came to the home, and on follow-up interviews at 6 and 12 months. We asked the women about their safety resources, use of safety plan, satisfaction with court, police and 38 EVALUATING DOMESTIC VIOLENCE PROGRAMS other systems, and about their continued exposure to violence (though it must be noted that this last factor is a controversial measure as far as women’s programs are concerned since a woman does not have control over her abuser’s behavior). The results from this study gave us a better picture of the impact that our program was having on the women’s lives and on the behavior of their abusers. How data is collected at the Domestic Abuse Project Almost since the creation of the Domestic Abuse Project, we have conducted data collection and analysis on a continuing basis. Our programs are better for having in place an efficient and economical system for collecting useful information. Besides providing ever-current data about our programs, this ongoing evaluation also provides us with historical records for comparison purposes. We usually collect our data through some type of follow-up survey. When men come into the program we inform them of the expectation that there will be follow-up. It is necessary to get written permission to contact the woman partner. When they leave the program, we get contact information. We inform women that they have the option of not participating. Samples of evaluation forms developed for use at the Domestic Abuse Project are located in the Appendix at the back of this manual. The intake through follow-up process is detailed for both abusers and victims/survivors, with a complete set of intake forms and summary sheets for each. Chapter 6 provides background about the development of the forms and suggestions for how they might be useful to you. MEASURES AND PROCEDURES 39 40 EVALUATING DOMESTIC VIOLENCE PROGRAMS 6 MODELING evaluation About the DAP sample MEASURES The sample measures included with this manual (see Appendix A and Appendix B) represent a great deal of effort over considerable time. They were developed section by section, some parts taking as long as a year to refine. The process generally began with DAP’s evaluation director and program administrator working together to develop a particular evaluation tool to measure outcome success, then seeking substantive input from the direct service staff to be sure the right questions were being framed. The contributions of direct service staff are essential in developing realistic measures. These measures are specific to the DAP program and can be helpful to your program if you customize them based on how your program works and what you have identified to be its goals, objectives, and outcomes. These measures are by no means perfect and we find ourselves continually revising them. We would appreciate your feedback on them as well. Using the Therapy Unit follow-up forms The first set of sample measures (Appendix A—Sample Measures I) are the forms used by the DAP Therapy Unit from intake through follow-up surveys. Summary forms are also included on which to summarize collected data. There are different forms for abusers and victims. In filling out both the abuser forms and victim forms, we usually prefer to interview the women partners of abusers, and not the abusers directly, because we depend on the women’s responses for greater accuracy. Detailed instructions that DAP staff follow in preparing and conducting interviews are included. To help assure victim safety, telephone calls are made by female interviewers. Names and addresses are gathered at intake and clients sign a release form advising them that follow-up will be conducted. An effort is made to keep up-to-date with address changes during the course of the program. Survey responses are collected through telephone interviews six months after clients have completed the program. Using telephone interviews to 42 EVALUATING DOMESTIC VIOLENCE PROGRAMS collect data involves a significant cost, but we have found it to be the only effective way to accomplish the task. Therapy clients are typically a very mobile population, and it often happens that six months after completing the program they have moved and cannot be located by mail. Funding for this type of follow-up is built into DAP’s grant proposals to its consistent funders and is included in research proposals. Besides being used for follow-up with program completers, these forms have also been used for clients who did not complete the group. The purpose of following up with program dropouts is to provide a measure of program effectiveness by comparing dropouts with those who completed the program. In measuring outcomes, we recognize that with domestic violence survivors the outcome can be part of a long process. They aren’t likely to be successful in leaving their abuser the first time they make an attempt at it. But each time they try, they may get closer to ultimately leaving if the abuser doesn’t change. Meanwhile, as a result of the program, they have many more options about how to be safe, and much more knowledge about how to recognize the escalations of violence so they can get out before the critical point. We recognize what we want our ultimate outcomes to be, and we identify measures that let us know we are getting there. We ask survivors to help us rate them on a scale so that we are not simply relying on our own measures of what those outcomes are. Instead, we rely on those who know best—the survivors of domestic violence. Using the Intervention Satisfaction Survey The Intervention Satisfaction Survey (see Appendix B—Sample Measures II) is a tool designed for DAP’s advocacy services. The purpose of the survey is to gain a sense of clients’ contact with advocates throughout the process from the time of an arrest to the time the court case is closed. The survey has only 15 questions and requires just a few minutes to fill out. The additional “possible questions” about the judge and court, and about orders for protection, were weeded out of the original survey. They were useful questions, but keeping the survey brief was a priority. The survey is given to a client when she has finished working with her advocate. At first the surveys were mailed to clients, but even with postage-paid return envelopes included, responses were few. Following up with telephone interviews was highly effective, but cost prohibitive. Finally, it proved effective as well as affordable to have the advocate hand MODELING EVALUATION 43 deliver the survey to the client at the time of their last contact, typically around pretrial or trial. The advocate explains that the survey is short and simple to fill out, and that it will let us know what is helpful about our program and whether improvements need to be made. The client is offered the option of filling out the survey at that time and depositing in the drop box at the front office, or she may mail it back in the business reply envelope provided. Using evaluation for systems advocacy The Intervention Satisfaction Survey (see Appendix B) also helps with DAP’s role in systems advocacy. Besides being asked to rate their experience with the system (quantitative measures), victims are invited to write comments (qualitative measures) about their experience in the system and about DAP’s services (see Survey questions 10 and 15). Many of the comments we have received have been invaluable in training judges and prosecutors. For example, a survivor commented that she was asked in court what she wanted done, but she didn’t want her abuser to hear her say that she wanted the court to “throw the book at him.” She felt that was not her job, that the court should throw the book at him for the crime he committed. Some judges frequently ask victims in open court: “What would you like us to do here—give him jail time, or . . .?” Now we use that comment to train judges about how they can do something differently to be more effective in the courtroom. We explain that survivors may not want to be put in the decision-making position, preferring that the court take that role. Victims may want to be able to say through their advocate that the abuser needs to be given an effective message, but they may not want to have to say it in court where they feel threatened or coerced by the abuser. We have received similar comments about police—“I want the police to make the arrest, not to ask me if I want him arrested.” In responses to Survey question 6—How happy were you with how the police responded to you?—if we find all of the victims from a certain precinct in our city are responding “not happy at all,” that is telling us there is a pattern in a particular precinct that needs to be looked into. This type of input helps us feed back to the system information about what they need to do to meet two goals we have for them: to be better at ensuring victim safety, and to be better at assuring abuser accountability. In addition to the Intervention Satisfaction Survey, another important source of feedback for DAP’s systems advocacy is the computer database explained in Chapter 9. 44 EVALUATING DOMESTIC VIOLENCE PROGRAMS 7 Analyzing, presenting, and USING results What did you FIND—and who cares? When you have collected all of your data, it is time to summarize it and figure out what it tells you about your program. The summary forms included with the sample measures in the Appendix are used to summarize evaluation data on a quarterly basis. Interpretation may not be easy. Results are often ambiguous. Learn to look beyond the surface for the real significance of your data. Whether positive or negative, your findings should be useful if the evaluation was carefully planned. At DAP, input from direct service staff is considered essential not only in the development of accurate measures, but for meaningful interpretation of results as well. As stated at the beginning of this manual, plans to disseminate and use your study findings should underlie the entire process of evaluation planning and implementation. What can positive results do for you? Certainly favorable evidence of program effectiveness affects funding decisions. And the efforts of staff are energized by knowing their work has a meaningful impact on people’s lives. Positive study results are also useful in outreach efforts directed at referral sources and potential clients. What if study results show that your program fell short of its goals? Learning that something in your program is not working might be among the most valuable results of your study. It can provide clear direction toward improving services and refining goals and objectives. Negative results can lead to positive change, and that is the goal of doing evaluation. Also consider that an unexpected result does not necessarily indicate failure to meet a goal. Maybe a particular measure wasn’t adequate to the task of answering your study question, or there could be a false assumption about what the result means. For example, the fact of significant 46 EVALUATING DOMESTIC VIOLENCE PROGRAMS numbers of women returning repeatedly to a shelter might be taken by some to indicate that the shelter program is failing to help women find safety. Some would term it recidivism, which is a standard focus of the criminal justice system. But further study might reveal that women’s use of the shelter is a positive indication that they are using their safety plans, which is probably one of your outcome objectives. Repeated shelter use may in fact be part of a long and rocky road to a safer life for these women and their children. Sometimes it is a matter of looking at a study result within a larger context. As another example, it is not unusual to find that women whose partners have completed a domestic violence program report increased threats of violence after the program than before. It may seem that a goal of decreasing the women’s exposure to threats has moved farther away. But the reality is likely to be the positive outcome that, as a result of their own program participation, the women have acquired increased sensitivity to the behavior of their batterers and greater skill in recognizing clues to escalating violence. It is important to be aware that some things you assume will change might, in fact, go in the opposite direction as a result of increased knowledge or other factors. (This example again highlights the problem of defining a decrease in women’s exposure to threats as a goal.) Sometimes there is a reluctance to undertake evaluation because of a fear of creating data about your program that you do not want people to have. At the Domestic Abuse Project, we feel it is better for us to be open and accountable, and to show that we are changing in response to what might be troubling findings. Our funders have appreciated our integrity. The programs of other agencies also benefit from the opportunity to learn from our experience, which is one of our purposes in making the information available. Honesty in reporting helps us move toward our overall mission of ending violence. How do you judge success? The same set of statistics about a program may suggest very different degrees of success depending on how you look at the numbers. In a DAP study of batterers’ programs that was done about a decade ago, the total number of men who had contacted us during the study period was 500. Appointments were made but many did not show up for intake. Fifty-seven percent, or 253 of the original 500 started a group. Of those who started, 100 dropped out sometime between the first and last ANALYZING, PRESENTING, AND USING RESULTS 47 group sessions. (These were 12- to 32-session programs.) And 153, or 31 percent, actually completed the program. Of those completing, approximately two-thirds were reported not violent at follow-up. PROGRAM DROPOUTS On what group do you base success? 500 called the program 100% 253 started the group 157% 153 completed the group 131% 2/3 reported not violent at follow-ups 120% based on Edleson & Syers (1990, 1991) It could be said that only one in five men who had contact with our program was not violent at follow-up. Not encouraging results, some would say. But we might judge the program’s full impact more accurately by the 153 who completed it, with about two-thirds of that group not violent at follow-up. Two out of three is an encouraging result. Another way to look at it is that two out of three not violent at followup is one good component of a community-wide response. It won’t solve everything, but it is a positive contribution toward one of the community’s overall goals. Other components in the picture are the shelters, crisis lines, coordinated interventions, domestic violence counseling, and advocacy services. Another study, in which we included program participants who received partial services, showed 50 percent of program dropouts not violent at follow-up. The group completing the program had approximately 15 percent fewer violent men at follow-up than the dropouts. This study addressed the question of whether results were actually due to the program or to other causes. This is a question researchers ask, and it is important to be prepared to answer it. 48 EVALUATING DOMESTIC VIOLENCE PROGRAMS Researchers might also call for a control group of people randomly assigned to no-treatment, but this is very difficult to achieve in domestic violence programs. Comparing results to dropouts and demonstrating consistent results over a variety of studies is a solid indicator of program impact without denying perpetrators of violence some contact with social services. Victim survivors are a priority audience for information on program impact. While “two out of three” can be considered a very good result for a specific social service program, the bottom line is that battered women should probably not rely on men’s participation in such programs for their safety. Conveying this fact honestly is being accountable for victim safety. Present your results and conclusions for maximum impact Most people in your audience—people with a stake in your program—will not be able to use every detail about your program that your evaluation produced. For each person or group that you present information to, highlight what they need to know in order to make decisions. Be able to say, “This is the most important thing to be found in the study.” Then make the simplest presentation that will account for the facts. For example, if the key information can be communicated in a simple bar chart, line chart, or pie chart, then make this the focus of your report. More complicated analyses can appear in an appendix, or be made available separately on request. The following set of charts for the imaginary “Superduper Community Awareness Program” is typical of the kind that can be produced easily with any of several popular computer programs. The different formats were created from the same data with just a few keystrokes necessary to change the appearance. There are many possible variations of each format. Note how different presentations emphasize different aspects of the data. ANALYZING, PRESENTING, AND USING RESULTS 49 Superduper Program Stats December 1996 January 1996 Aware 11% Not Aware 89% These line graphs show the same program data as in the pie charts, but played out month by month. Line graphs emphasize rate of change over a particular time period more than amount of change. In this example, a specific activity (a public service campaign) is highlighted by relating it to a dramatic increase in change. In this pair of line graphs, note how relative amounts of change from month to month is either emphasized or downplayed depending on how many unit divisions are used on the vertical axis. 50 Not Aware 38% Aware 62% Pie charts help you to compare parts against each other and against the whole. The first chart represents the level of community awareness before the program and the second shows the difference at the end of one year. Using different colors or patterns for the various “slices” adds to the visual effect. Placing the slice that you want noticed toward the lower side of the pie (appearing to be nearest the viewer) will emphasize it. 1996 Program Awareness Stats 70 60 50 Public service campaign> 40 30 20 10 0 J F M A M J J A S O N D O N D 1996 Program Awareness Stats 100 50 0 Public service campaign> J EVALUATING DOMESTIC VIOLENCE PROGRAMS F M A M J J A S 1996 Program Awareness Stats 70 60 50 Public service campaign> 40 30 20 10 0 J F M A M J J A S O N D Bar charts can be vertical or horizontal and two-dimensional or threedimensional. They show changes over a particular time period or help you to compare totals for several data series. The amount of change appears more significant when the vertical axis stops at a percentage just above the highest bar (or when it starts at a percentage just below the top of the lowest bar). In these examples, the relative change would appear somewhat less significant if the vertical axis were continued to 100 percent. 1996 Program Awareness Stats 70 1996 Program Awareness Stats 60 50 40 Public service campaign> 30 20 10 0 J F M A M J J A S O N ANALYZING, PRESENTING, AND USING RESULTS D 51 More ways to present data Mean (or average), median, and mode are statistical tools for interpreting data. They are examples of the many different ways of presenting information, depending on the needs of your audience. Each says something different about a particular set of data. Since there are likely to be people in your audiences who don’t have need to use such terms regularly, it is a good idea to include brief definitions: ❿ Mean—Same as the average. Obtained by adding together all the data (such as scores) and then dividing by the number of clients to arrive at the statistical balance point of a distribution. Even a single extreme measure at either end of the scale will skew the average. In such cases, either the median or mode is likely to give a more realistic picture. ❿ Median—The middle measure (such as a score) in a distribution so that half the measures are above and half below it. In an even number of scores, compute the average of the two middle scores to obtain the median. ❿ Mode—The category with the largest number. For a questionnaire item that clients are asked to rate on a 1 to 5 scale, the number that occurs most frequently in the responses is the mode score. ILLUSTRATION OF MEAN, MEDIAN, AND MODE QQ QQQ Q QQQQQ 52 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Mean is the average. ❿ Median is the point where half are above and half are below. ❿ Mode is the category with the largest number. Preparing your interpretation of results for your audience Have a particular person or group in mind as your prepare your presentation. Too often people cannot use the information presented to them because they don’t adequately understand it. Practice on staff first. Their honest feedback will help you be more effective. ❿ Begin by making a clear statement of your goals and objectives. Set them as the criteria for success against which your results are to be compared. ❿ Present your main points in different ways. Remember that some people learn best from charts, others by numbers, and some by reading or hearing a story that illustrates your finding. ❿ Label tables and charts so they can be interpreted independently from the text of the report. Many users of reports will page directly to the table or chart and base their judgment on their understanding of it. ❿ Present results in order of importance. Emphasize the most important results at the beginning. Give less space to less significant results, presenting them as supplementary to the primary results. ❿ Separate results from conclusions. A good report presents the results first, then draws conclusions from them. ❿ Involve others in drawing conclusions from results through a series of consultations. When preliminary results are available, present them to a variety of people having an interest in the program. Carefully consider their ideas and conclusions regarding the results before composing the final report. ❿ In your presentation, address alternative interpretations of the results and make a case for the various recommendations they might support. There are likely to be people who will attack your results if they think there is another explanation for them. Your results and conclusions will carry more weight if it is apparent that you have considered varied views regarding the results. ❿ State conclusions in a way that enhances the likelihood of their being used. Anticipate questions your audience will be asking themselves and then draw conclusions that will be useful in answering those questions. This focus on the concerns of the people who have an interest in your program should be carried through the entire process, from the initial decision to conduct evaluation through the final presentation of results and conclusions. ANALYZING, PRESENTING, AND USING RESULTS 53 Disseminating results—a DAP example Disseminating results to the larger public is a priority at the Domestic Abuse Project. It is part of our mission to provide effective model programs. This requires us to continually evaluate our services and implement changes if evaluation results warrant. It is our belief that sharing these study results and program changes with the larger community will enable other programs to plan and implement positive changes based on our research findings. These “shared results” extend the scarce resources that are available to programs countering domestic violence. Spreading the word about a new study—When a recent DAP study was completed on children’s use of specialized services, a carefully planned publicity effort went into action. A news conference was called and three hundred invitations were sent to professionals who are in a position to impact the issue. People invited included those who work in the medical services, child protection, child guidance centers, other domestic violence agencies, and city and county elected officials. News releases were sent to media representatives throughout the metropolitan area. (Reprints of the invitation and news release begin on the next page.) Forty people—considered a good turnout—attended the two-hour conference to hear a presentation of the findings of the study and ask questions of the researchers and those implementing the changes to the program as a result of the study. In the days following, television news features and newspaper articles announced the study, explained its implications, and provided the public with important information about the specialized services available to child witnesses of domestic violence. DAP’s Training & Research Update—In addition to local distribution of study results, DAP also publishes the Training & Research Update, a newsletter-style report distributed worldwide to professionals in the domestic violence field and related fields. The Update presents highlights of study results and provides ordering information for those who wish to obtain full reports. The February 1997 issue featured a summary of the children’s study mentioned above. (The article is reprinted on the following pages right after the sample news conference invitation and media release.) 54 EVALUATING DOMESTIC VIOLENCE PROGRAMS Sample invitation—DAP’s news conference invitation is reprinted below. Agency letterhead is always used for public correspondence. A signature by the agency’s top administrator helps to convey the importance of the message. DAP DOMESTIC A B U S E PROJECT [date] [name, position, and address of invitee] Dear [name of invitee], I am writing to invite you to an upcoming presentation and discussion regarding the accessibility of domestic violence services for children who witness violence in their homes. I am eager to share with you the results of a three-year DAP study which was funded by the Phillips Foundation. This study of almost 200 children and their families was aimed at improving the accessibility of children’s domestic violence services and determining why some parents enroll their children in specialized services while others do not. We found the results to be insightful and want to share them with others like you who are striving to make a difference for families in our community. Please join us on Tuesday, June 10, from 9 a.m. to 10 a.m., at [location]. Jeffrey Edleson, PhD, and Einat Peled, PhD, the two primary researchers for this study, will discuss how the study was conducted, and more important, what we have learned as a result of the research. DAP’s children’s therapists, Jerry Jensen, MA, LPP, and Shannon Schmidt, MS, LP, will also provide a brief overview of our current children’s program, specifically detailing the changes implemented since the completion of the study. Dr. Edleson has directed DAP’s Research and Evaluation Unit for more than 14 years. He is also a Professor in the University of Minnesota School of Social Work and the Director of the Higher Education Center Against Violence and Abuse. He has published more than 60 articles and several books on domestic violence, groupwork, and program evaluation. Dr. Einat Peled is an Assistant Professor in the Bob Shapell School of Social Work at Tel Aviv University, Israel. She was a researcher at the Domestic Abuse Project at the time of this study. She has conducted research and published several articles and books on battered women and child witnesses both in Israel and in the United States. Her work includes DAP’s manual for working with child witnesses, Groupwork with Children of Battered Women, which she coauthored with Diane Davis, MA, SP, LGSW, Director of Therapy at DAP. I hope you can join us for part of your morning on Tuesday. A continental breakfast will be provided. I look forward to seeing you and continuing to work with you on the issue of domestic violence. Sincerely, Carol Arthur Executive Director 204 West Franklin Ave., Minneapolis, MN 55404 • Tel. (612) 874-7063 • Fax (612) 874-8445 • E-Mail: dap@mndap.org ANALYZING, PRESENTING, AND USING RESULTS 55 Sample news release—Prior to the news conference, the following news release was sent on agency letterhead to newspapers and television and radio stations throughout the metropolitan area. Generous line spacing is preferred for news releases. DAP DOMESTIC A B U S E PROJECT [date] Contact: FOR IMMEDIATE RELEASE Carol Arthur, Executive Director Ann Kranz, Director of Marketing (612) 874-7063 Domestic Abuse Project releases new study on children’s services The Domestic Abuse Project is sponsoring a presentation and discussion regarding the accessibility of domestic violence services for children who witness violence in their homes on [date], 9 a.m. to 11 a.m. at [location]. DAP will release the results of a three-year DAP study which was funded by the Phillips Foundation. This study of almost 200 children and their families was aimed at improving the accessibility of children’s domestic violence services and determining why some parents enroll their children in specialized domestic violence services while others do not. Jeffrey Edleson, PhD, and Einat Peled, PhD, the two primary researchers for this study, will discuss how the study was conducted, and more important, what was learned as a result of the research. The Domestic Abuse Project’s children’s therapists, Jerry Jensen, MA, LPP, and Shannon Schmidt, MS, LP, will also provide a brief overview of DAP’s current children’s program, specifically detailing the changes implemented since the completion of the study. Dr. Edleson has directed DAP’s Research and Evaluation Unit for more than 14 years. He is also a Professor in the University of Minnesota School of Social Work and the Director of the Higher Education Center Against Violence and Abuse. He has published more than 60 articles and several books on domestic violence, groupwork, and program evaluation. Dr. Einat Peled is an Assistant Professor in the Bob Shapell School of Social Work at Tel Aviv University, Israel. She was a researcher at the Domestic Abuse Project at the time of this study. She has conducted research and published several articles and books on battered women and child witnesses both in Israel and in the United States. Her work includes DAP’s manual for working with child witnesses, Groupwork with Children of Battered Women, which she coauthored with Diane Davis, MA, SP, LGSW, Director of Therapy at the Domestic Abuse Project. ### 204 West Franklin Ave., Minneapolis, MN 55404 • Tel. (612) 874-7063 • Fax (612) 874-8445 • E-Mail: dap@mndap.org 56 EVALUATING DOMESTIC VIOLENCE PROGRAMS REPRINTED FROM TRAINING & RESEARCH UPDATE, NO. 9, FEBRUARY 1997, DOMESTIC ABUSE PROJECT, MINNEAPOLIS, MINNESOTA Why children don’t use specialized services New study focuses on programs for children who have witnessed abuse by Jeffrey L. Edleson, PhD, and Einat Peled, PhD A DRAMATIC INCREASE in consideration of children who witness violence in their own homes is evidenced by new public policy analyses, a growing body of published research, and innovative social services. Intervention is still in its infancy, and there has been little study of the degree to which these children use the new services or of the factors that influence their access to services. A recent study by the Domestic Abuse Project (DAP) in Minneapolis, Minnesota, revealed significant underuse of services designed for children who witness domestic violence. Specific barriers to access were identified by the study. This is a summary of the DAP study results. Those wishing more detailed information are encouraged to use the enclosed order form to obtain copies of the full research report from DAP. The report is in two parts (see last two items on order form check list). One paper focuses on quantitative predictors of child participation and completion (Peled & Edleson, 1997); the other focuses on qualitative comments made by parents about obstacles to their children’s receipt of services (Peled & Edleson, 1996). Study showed few eligible children used services The study sample consisted of 194 children (43 percent boys and 57 percent girls) between the ages of four and 18. The children were from 97 families. More than half the children lived only with their mother; 24 percent lived with both parents; 12 percent lived only with their father; and the rest were in the care of relatives, in foster care, or in other arrangements. The first purpose of the DAP study was simply to describe children’s use of services. Informal observations were confirmed that only a small number of the children related to adult clients were making use of children’s services. The data, as indicated in the chart, showed that less than a third of the children ever used available services and even fewer continued through program completion. No contact with agency 69% Intake only 4% Participated Completed in group program 6% 22% In addition, more than a third were reported to have received some type of services from another program, most often provided through a mental health agency or school-based program. Mothers had major role in level of children’s participation Included in a logistic regression were variables highly associated with children’s service participation: child’s age; child’s living arrangement; last incident of violence directed at the mother; services received by both mother and father; severity and type of violence against the mother. Only two variables—child’s living arrangements and mother’s service level—entered the regression equation. Results indicated that: ❿ Children’s participation in domestic violence services was most likely to occur when a child lived only with his or her continued on next page ANALYZING, PRESENTING, AND USING RESULTS 57 REPRINTED FROM TRAINING & RESEARCH UPDATE, NO. 9, FEBRUARY 1997, DOMESTIC ABUSE PROJECT, MINNEAPOLIS, MINNESOTA continued from previous page mother. In the study, children who lived with only their mothers were more than twice as likely as other children to participate in services. ❿ The more services the mother received, the more likely her child was to participate in agency programs. Children whose mothers participated only in a women’s group at the agency were almost two-and-a-half times more likely to participate in the children’s program than were children whose mothers received no agency services. Children whose mothers participated in both a women’s group and received other agency services were again almost two-and-a-half times more likely to participate in the children’s programs than were children of mothers who only participated in the women’s group. These results are no doubt influenced by the fact that many more children in this sample were living with single mothers than in other arrangements. Other factors likely to have influenced the outcome: ❿ Women generally come to this agency voluntarily, whereas men overwhelmingly come to services through court or other social mandates. ❿ The potential harmful effects of violence on children and the availability of children’s services are often discussed with mothers as part of the agency’s support groups for women. Similar discussions occur on an irregular basis with participating men. ❿ The more services a woman receives, the more likely she is to be exposed to information on children’s services and to have help in making decisions about her child’s service needs. ❿ Services beyond the group programs are likely to include a parenting group, often conducted parallel to the children’s groups. ❿ Parent group members were exclusively mothers, making the highest level of services to mothers likely to include parent 58 EVALUATING DOMESTIC VIOLENCE PROGRAMS groups held in conjunction with the children’s program. ❿ There is some indication that mothers, rather than fathers, are also primarily responsible for child-referrals. Interviews reveal barriers to children’s access to services Extensive telephone interviews with parents brought out six major obstacles to children gaining access to services: ❿ Technical barriers—Some parents said that transportation difficulties prevented them from getting their children to the program. Others cited the group schedule, explaining that rushing their children from school to the agency for afternoon sessions was a burden. ❿ Violence-related stress—Scheduling was even more difficult for recently separated parents. Newly separated mothers, often experiencing dramatic decreases in standard of living, had to cope with single parenthood, and sometimes continued to be threatened or subjected to new violence by their former partners. Child programming concurrent with adult groups and weekend schedules were suggestd by some parents. ❿ Perceived child needs for services—Many parents didn’t see service as appropriate for their child. Some lacked complete or accurate information about the services available, or misunderstood the criteria for admitting their child to the program. For example, some thought groups were only for children who had themselves been physically abused. Others didn’t understand the goals or nature of the program. ❿ Parental opposition to child’s participation—Even well-informed parents sometimes opposed enrolling their child in services, expressing mixed feelings about reopening the topic of violence or exposing continued on next page REPRINTED FROM TRAINING & RESEARCH UPDATE, NO. 9, FEBRUARY 1997, DOMESTIC ABUSE PROJECT, MINNEAPOLIS, MINNESOTA continued from previous page their child to others’ even more violent stories. A number of fathers denied their children were ever exposed to violence. Sometimes mothers also defined witnessing in a narrow way that denied or minimized the impact that witnessing violence has on their children. ❿ Agency’s lack of acceptance of men’s parenting role—At the time of this re- If we build children’s services, we cannot just assume that “they will come” search, men who participated in DAP’s program could not refer their children to the agency’s programs. This policy has since changed. Throughout the interviews, it was clear that men continued to have contact with their children. Few understood the effects that witnessing violence has on children. Some wished to have their children receive services. Some men and women expressed a desire that the agency take a more active role in engaging fathers. ❿ Child opposition to participation and related program initiated changes—The child’s own behavior was the focus of a number of factors affecting child participation in services. Some children expressed opposition to the group. Others were assessed by staff during intakes or group sessions to require individual counseling or other services. Group services were not appropriate for all children. Study indicates need for increased attention to both mothers and fathers The study data showed that service usage by children depends a great deal on the mother’s own participation in services. A battered woman’s technical (e.g. transportation, scheduling) and emotional needs require greater attention if she is to support her child’s healing process and maintain his or her participation in children’s services. In addition, it seems clear from DAP’s experience that fathers are often overlooked as a way to reach unserved children. Domestic violence programs should not reinforce traditional social teachings that dismiss men’s responsibility for their children. Certainly caution must be taken to ensure that children are not pawns for batterers seeking additional means to manipulate and control their partners. But greater attention should be directed to educating men who batter about the effects of violence on their children and how their children might benefit from support and education around issues of violence. While a great deal of attention is now being focused on designing and providing innovative services to these children, we should not overlook the significant hurdles that parents and their children face in using such services. If we build children’s services, we cannot just assume that “they will come.” We also have to pave the way for children and their parents to access these services. ANALYZING, PRESENTING, AND USING RESULTS 59 Exercise Putting your results to use ❿ Who will receive the results of your evaluation? Write down the various groups that have an interest in your results. For each audience listed, explain which results will be of most interest to that group. ❿ For each audience that you listed, describe the best way to present results for that group. ❿ What obstacles (situations or people) may prevent your final report from being used? For each obstacle you list, write down a suggestion for overcoming it. 60 EVALUATING DOMESTIC VIOLENCE PROGRAMS 8 Putting it all TOGETHER Guided exercise in developing an evaluation PLAN The time has come to combine your work from the previous chapters and describe a comprehensive plan for your evaluation. Your agency may have multiple programs, and each program may have multiple components. It is wise to evaluate all parts of your programs, but for this guided exercise it will be most useful to focus on a single program or component of one program. ❿ Identify here the program or program component you are currently evaluating or one that you intend to evaluate. ❿ Collect and review any documents about the identified program that describe its mission, its goals, and the outcomes it is to achieve through the services or activities it carries out. For reference, list those documents here. 62 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Collect and review any reports of prior evaluations of the identified program or similar evaluation at the same organization. Describe them here. ❿ Talk to people associated with the program and ask them what they see as the mission and goals of the program, and what outcomes they think should be achieved. The people you talk to should feel free to diverge from what the formal documents state. Include a variety of people who have contact with the program, such as staff, board members, clients (for men’s programs also be sure to include victims/survivors). Jot down notes of their responses. PUTTING IT ALL TOGETHER 63 ❿ Taking into consideration the research you conducted in the previous three steps, list your program’s goals. For each goal, consider all the different outcomes that might reflect change, then list each goal’s outcome objective(s). In stating outcome objectives, try to include the elements of time and increase/ decrease in terms of percentages or numbers. Be realistic about what number or percent can be achieved in a given time. A sample wording of an outcome objective is as follows: SAMPLE EXPRESSION OF AN OUTCOME OBJECTIVE The agency will train 10 new police cadets this year. Six months after training, at least 80 percent of these cadets will rate their training “very useful” in helping them to understand the dynamics of domestic violence and at least 90 percent will feel “better prepared” to respond to domestic assault calls because they understand police protocol and current law as a result of the training. The first time you set goals they may not be realistic. And, given changes in welfare laws and other factors, reality may seem to keep changing. Don’t expect to have total control over what you can accomplish. The important thing is to set a direction for your program. Use the space below and on the next page to list your program goals and outcome objectives: 64 EVALUATING DOMESTIC VIOLENCE PROGRAMS (Use this space to continue your list of goals and objectives) PUTTING IT ALL TOGETHER 65 ❿ For each outcome objective listed in the previous step, list one or more ways of measuring its achievement. For each measure, specify where you will get the data, and when you will collect it. 66 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Indicate what program resources will be needed to carry out this measurement plan. PUTTING IT ALL TOGETHER 67 ❿ Describe here any therapeutic, ethical, practical, resource, and skill issues that you foresee in relation to your evaluation plans. List ways that such obstacles might be overcome. 68 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Describe how you will ensure that your evaluation results get used. PUTTING IT ALL TOGETHER 69 ❿ For the final step in your evaluation plan, describe how you will present or disseminate your results to everyone who has an interest in your program. Don’t underestimate the number of people who might be interested in what you have learned. You are reading what we learned right now! 70 EVALUATING DOMESTIC VIOLENCE PROGRAMS 9 Meeting future INFORMATION needs Do you need a COMPUTERIZED database? Program evaluation is a major undertaking with many potential benefits. The value of your efforts is further increased when the information you collect about your programs can be made to serve multiple purposes. Indepth information about programs collected over a span of years can become a powerful tool for long-term decision making. To be useful, the information should be well organized and readily accessible for a variety of purposes. A computerized database is a good way—many would say the only way—to manage large amounts of information. In addition to providing for the current and future information needs of your own organization, a computerized database can be an important resource for academic research in the field of domestic violence, adding momentum to the drive toward the overall mission of ending violence against women. At DAP, our computer database is also a tool for criminal justice systems advocacy. We use it to track what the system is doing with domestic violence cases. In Minneapolis, for more than a decade DAP has been the only source of information to the system about what happens to cases, though that is changing now that prosecutors and police are beginning to look at keeping computer databases. We have found that the system is not responsive to social service organizations without the kind of concrete data that a database provides. If we attempt to convince the system of a problem using only the stories of a few victims, our evidence is dismissed as “anecdotal information.” Using our computer database, DAP was able to feed back to the system the detailed information that in the city of Minneapolis, in a year, out of 4,000 cases in which a domestic assault arrest was made, only 900 got through the system with a guilty plea or finding. We can then ask for an accounting of what happened to the other 3,000-plus cases and what was done for the victims in those cases. The police, prosecutors, and judges see value in the information we can provide through our database, and as a result they are more willing to discuss the issues of domestic violence. 72 EVALUATING DOMESTIC VIOLENCE PROGRAMS What is a computerized database? Even if you or others in your organization have not previously worked with computerized databases, there is no reason to be intimidated by the idea. Everyone has created and worked with databases of one kind or another. A database is simply pieces of related information stored in a central location where it can be used, easily or otherwise, by one or more people. A shoebox full of recipe clippings or tax records, a personal address book, the filing cabinets at the office—all these are databases. When you put all your tax deductible receipts in a separate envelope in your shoebox, arrange your recipes by main ingredient, add fax and e-mail information to all the names in your address book, or open a file drawer to look up some information in a particular folder, you are managing your database. These are good ways to manage small amounts of data. In general, a computerized database is most helpful when you are working with large amounts of information that you want to use for more than one purpose. Information is entered into a computerized database only once. You can then use computer applications (programs) to select and organize your information for many different purposes. Much staff or volunteer time can be saved by using the database for tasks such as producing routine reports and answering questions that funders and others might ask about your programs, in addition to preserving information for future use. This chapter covers some of the basic considerations that might enter into your decision about setting up a computerized database, followed by an overview of the Domestic Abuse Project’s legal advocacy database. It is possible to design and create your own computerized database using one of the many popular programs on the market. Another approach is to work with a professional database developer who will design a customized system that you and your staff will be able to maintain and use. Before trying to decide who should develop your database, give some serious thought to what kind of information you want to store in a database and what you want to be able to do with that information. The nature of your information needs will be a factor in whether you will be able to create your own database or will require the services of a professional developer. MEETING FUTURE INFORMATION NEEDS 73 How complicated or detailed is your information? On your desk you might keep a card file of people you call frequently. Probably each name with its related information occupies a separate card, and the cards are arranged alphabetically so you can find information quickly. This is the structure of your phone list database. Computerized databases also need to be structured. Just as your phone list is made up of cards that may each contain several lines of information, a computer database is made up of records, each having several fields of information. If you were to enter your phone list into a computerized database, you would discover that the computer requires a very rigid structure. On your desktop card file you have no trouble recognizing what each line of information on a card represents. You would never mistake a telephone number for a person’s name even if they were written in a different order than on other cards. Computers cannot recognize information the way we do, so a very explicit structure is required. Each field in a record represents one meaningful piece of information. Choosing what should go in a field is not always easy. In a first attempt at creating a database from the card file mentioned earlier, a beginner might decide to have each record include four fields representing each line on the card: name (first and last), street address, city/state/zip, and phone number. However, when the need arises to sort the information by last name or state, under this structure it would be cumbersome at best and impossible at worst. The computer will handle the fields containing first/last names or city/state/zip as single pieces of information, no part of which can be isolated easily. To allow the task to be accomplished, first name, last name, city, and state/zip—and any other items by which you want to sort—should be defined as separate fields. The structure of your computerized database (that is, how the fields and records are defined) will largely determine what you are able to do with the information you have stored in it. Learning to define meaningful pieces of information is an important aspect in creating a useful database structure. If your information is very detailed, you may well benefit from the services of a good professional developer. For example, suppose your unit of information is each domestic assault arrest. For each record stored in the database, you might want to include name, address, phone number, demographic information for victim and perpetrator, and information about the progress of the case through the courts. With that high level of detail, defining useful fields is a challenge. The expertise of a professional developer can be important. 74 EVALUATING DOMESTIC VIOLENCE PROGRAMS Do you need a “flat” or a “relational” database? The structure of a computerized database could be pictured as a single table of information. The horizontal row across the top might contain category headings (fields) for client names and all the information you have collected or plan to collect about the clients in a program. All the spaces in the body of the table are filled with the specific details, arranged under each heading, that relate to the individual clients. Each horizontal row is a record. The computer program that lets you navigate this table is called a database manager. This program lets you manipulate your data with a variety of commands, the primary ones being add, sort, search, print, edit, and delete. Many of the inexpensive database products on the market are designed to handle only a single table (database). This type is called a “flat” database manager, or just a flat database. You might find it useful to have more than one database, each handling a different kind of information. A “relational” database manager lets you search multiple databases simultaneously, relating the data in the various tables in complex ways to meet a variety of purposes. Some of the databases might even be at remote locations. A relational database manager is a type of programming language. It contains a variety preprogrammed commands for functions such as storing data in tables and responding to search queries. Should you create your own database? You and your staff have the best understanding of the information you are working with and the uses you need to make of it. You might consider creating your own database if there is someone in your organization who has some experience with database software, is skilled in learning new computer applications, and can be allowed sufficient time to develop the database. Even if you expect that your database will be developed and maintained by one person within your organization, it is important to have a plan for how that person will educate others in the organization about how to use the database. It’s always risky to have an important function understood by only one person. This is especially a concern with a database because it requires an understanding both of the types of information contained in it and its uses within the organization, in addition to the technical elements of the database that allow you to use that information. MEETING FUTURE INFORMATION NEEDS 75 What to look for in a database management product In evaluating a particular database management program, consider whether it has the right set of features for the tasks you want to accomplish. It should have the capabilities you need for your purposes, but you don’t want to pay for specialized features that aren’t relevant to your needs. A good way to gain an understanding of what the market offers in database programs is to read product reviews in the computer magazines that you can find on almost every magazine stand. Most of the computer magazines also have Internet Web sites where you can read and download product reviews from past issues. Following are some qualities to consider in addition to data-handling capabilities. Of course, the advantages of each level of refinement have to be weighed against your budget. ❿ Ease of use—To input and manipulate data, you choose from a variety of commands presented through a series of screens. These screens are called the graphical user interface or GUI. If many staff and volunteers will need to use the database for a variety of purposes, a user-friendly GUI is an important consideration. Screens that are well-designed are attractive to look at and easy to understand and use. There should be built-in safeguards against putting data where it doesn’t belong or leaving out essential data. This decreases the amount of time needed to train users and reduces the chance that an inexperienced user will inadvertently enter incorrect data. If only one person will work with the database, these kinds of enhancements are less critical. ❿ Speed—If the database is large, with highly detailed information, it could take a long time for it to respond to your commands. In that case, speed of response might be a purchasing factor. ❿ Programming ease—If you plan to create your own database, find out if the product is easy or difficult to program. ❿ Technical support—Inquire about availability of technical support. With the purchase of many computer software products, some amount of technical support is available from the software manufacturer if you need assistance in using the product. Some manufacturers are better than others at supporting the products they sell. Usually, you are given a phone number that will connect you with a technician who is familiar with the product you have purchased. Technical support calls are usually not toll-free, and long distances charges can build up quickly when trying to sort out problems. Other than telephone charges, the support 76 EVALUATING DOMESTIC VIOLENCE PROGRAMS is usually made available at no charge for a certain period such as 30 days. After that, charges may apply. Many computer products also offer general technical support through the manufacturer’s Internet Web site. Should you hire a professional database developer? If your organization would benefit from a computerized database and your information needs are fairly complex, or no one in the organization is available to take on the task of setting it up, you might want to consider working with a professional developer. ❿ Communication—Choose someone who communicates clearly with you. It’s easy to be intimidated by technical expertise, especially if you’re a computer novice. Trust your feelings. If you are uncomfortable asking questions of a developer, or if you don’t understand his or her answers, it’s unlikely that the developer will produce a database that helps your organization to do its work more effectively. ❿ Technical support—The developer should be available to provide ongoing technical support after initial setup. You will inevitably have questions as you become familiar with using the database. If a potential developer is unable to commit to a long-term involvement, she or he may be able to connect you with another individual or organization that could provide your ongoing support. Address these issues in your initial discussions with any developer whose services you are considering. ❿ Cost—Funders are increasingly recognizing the importance of technology for nonprofit organizations. One-time development projects are sometimes the preferred focus of funders rather than general operating funds. It is important to clearly demonstrate to a potential funder how the database will help your organization achieve its mission. Also consider the possibility that volunteering or working at a reduced fee to develop your database can give a professional developer the satisfaction of working on your organization’s mission. It can also provide them an interesting technical challenge. A small nonprofit organization might offer the developer more flexibility in using technical skills than is usually encountered in working with larger organizations. MEETING FUTURE INFORMATION NEEDS 77 What information should you include in your database? Whether you create your own database or work with a professional developer, you will need to specify what information will be in it. The database allows you to quickly sort, select, and organize the stored information. If, for example, you want to know how many of the people you serve are between 25 and 34 years old, you need to include birth dates in the database. It seems obvious, but it is surprisingly easy to overlook collecting and entering pieces of data that will be critical to answering your questions. Try to think of every question that you might want to use the database to answer. It might seem logical to store in the database any related piece of information you can obtain. But there is a cost to that in terms of the time required to enter the information, in the amount of computer disk space required to store it, and in the speed of response of your database. A good rule is to include information that you want to manipulate (sort, sum, select). Information needed solely for reference may be better stored in your paper files. Before developing a complicated database, consider how much of the information you plan to include is under your control. If there is information you would like to include that may become unavailable to you because of a change in law or a change in a relationship with an individual or organization, it might be a good idea to design your database so that it will still be useful even without that information. DAP’s legal advocacy database The screens that we use to enter data into DAP’s court database and final disposition database are reproduced at the end of this chapter. ❿ Purpose—The primary purpose of the database is to collect and analyze data that permit evaluation of the effectiveness of the criminal justice system response to domestic violence. As it is currently designed, it doesn’t serve as a good tool to evaluate the effectiveness of DAP’s legal advocacy services. ❿ Data—The database includes all cases in which Minneapolis police went to a scene and filed a police report indicating misdemeanor domestic assault, whether or not police made an arrest. The database also includes the court outcomes for all Minneapolis misdemeanor or domestic assault arrests. 78 EVALUATING DOMESTIC VIOLENCE PROGRAMS ❿ Uses—An individual case can be tracked from initial police involvement through final court disposition. More important, system-wide statistics can be compiled to allow comparisons over time, between different areas of the city, among different racial groups, etc. So far, the database has been most helpful in allowing DAP to feed back to the criminal justice system (or parts of it such as police, judges, etc.) information about how response to domestic violence has changed over time—for example, whether arrests have increased, whether the percentage of defendants convicted has changed, whether more or fewer abusers are being mandated to treatment, etc. The greatest potential for the database may lie in giving DAP the ability to address ad hoc questions such as how the conditions of sentencing from Judge “X” compare with the total of all judges. Another example is the “Top 50 Abusers” report that DAP prepared by selecting from the database those defendants with the most domestic assault charges over a limited time span such as a year. This was effective in illustrating for the criminal justice system that a response of minimal or no consequences for early offenses often leads to more frequent and more severe offenses. ❿ Data availability—Most of the data in the legal advocacy database is public and is obtained from criminal justice system sources. A small amount of the data is obtained through DAP (such as information about victim/survivor satisfaction with police response). Although much data on arrests and criminal court proceedings is public, that does not equate to readily and cheaply available. DAP is currently completing a project that will create an electronic interface between the Minneapolis police computer system and DAP’s database. DAP has a long-standing collaborative relationship with police and prosecutors and a contract to provide advocacy to Minneapolis victims/survivors of domestic assault. Nevertheless, it has taken four years to develop this project. Programs in smaller jurisdictions might be better served by obtaining data on paper, although there is often a handling charge for that data which may be prohibitive for smaller programs. MEETING FUTURE INFORMATION NEEDS 79 Legal Advocacy: Court Database DAP’s court database screen is represented below. Court Data Release Final Defendant Victim Witness Disposition Find New Save Close Case Ctrl No: Sip Case No: Sip Person No: Amended Charges: Court Date Next: Next Court Level: ❷ Judge Final Disp: Court Level Final Disp: Bail Amount : Final Disp Date: ❷ Judge Cond. of Release: Pros Atty.: COR Court Level: Explanation: Case Control Number: Use this number to track a client. By using this number, you can figure out their history. For example, in #MP96123456, MP stands for male perpetrator and 96 stands for the year the assault happened. The numbers following are taken from the SIP number. SIP Case Number: SIP stands for “Suspect In Progress.” It is used by Hennepin county to track the case as it goes through the court system. SIP Person Number: Like the SIP case number but used to track individuals. Amended Charges: This is to note any changes in the charges that were made during the court case. For example, a fifth degree assault charge might be amended to a DOC (disorderly conduct). This is where that change should be noted. 80 EVALUATING DOMESTIC VIOLENCE PROGRAMS Court Date Next: Next court date. Next Court Level: This lists what type of a court appearance the next hearing will be. The different codes to choose from are as follows: Next Court Level/Court Level Final Disposition A&D A&D APP Appearance APPBW AppBw ARRN Arraignment FILED Filed By CA PT Pre-Trial T Trial Bail Amount: Dollar amount is listed here. Judge Final Disposition: Name of the judge that tried the case. Final Disposition Date: This list the next court date. Court Level Final Disposition: See next court level. Prosecuting Attorney: Name of the prosecuting attorney. MEETING FUTURE INFORMATION NEEDS 81 Court Data: Final Disposition DAP’s final disposition database computer screen is represented below. Final Disposition Disposition Codes Final Disposition Disposition Codes for the above data base are listed below. They are entered on the left side of the datebase.Under Final Disposition, notes regarding each code are to be entered.Using a jail sentence as an example, “other” would be entered under Disposition Code. Information such as number of days of the jail sentence, the days served, and days left to be served are entered under Final Disposition. Charges: Charges appear in a user field above the disposition database. CFD DIS GF GP GSOI MODBC NGF UN Continued For Dismissal Dismissal Guilty Finding Guilty Plea Guilty Stay of Imposition 609.135 Motion of Dismissal by City Attorney Not Guilty Finding Unknown Disposition Code CD DV MH NSS 82 Chemical Dependency Treatment Domestic Violence Treatment Mental Health Treatment No Same or Similar EVALUATING DOMESTIC VIOLENCE PROGRAMS NVC NVOFP PROB1 PROB2 OTHER REST No Victim Contact No Violation of Order for Protection Probation 1 Year Probation 2 Years Other Restitution MEETING FUTURE INFORMATION NEEDS 83 84 EVALUATING DOMESTIC VIOLENCE PROGRAMS GLOSSARY Activity—The services provided by a program to its clients. Activities that might be measured include numbers of crisis calls taken, group sessions delivered, family or individual counseling sessions, community speeches given, etc. Advocacy—One possible definition is: individual and system change work on behalf of safety for women. A narrow interpretation focuses on legal advocacy on behalf of battered women, especially in the criminal courts. Audience—All the people who have an interest in your program and to whom you will present your evaluation results and conclusions. Examples: clients (men, women, children), program staff, board of directors, funders, community, policymakers, police, courts, medical establishment, social services, etc. Average—Same as the mean. Obtained by adding all the measures (such as scores) and then dividing by the number of measures to arrive at the statistical balance point of a distribution. Even a single extreme measure at either end of the scale will skew the average. In such cases, the median gives a better picture of what might be considered a typical score. Clients—Direct recipients of program services. Conclusions—Interpretive statements made about the results of the research. Data—Information collected in a systematic way and used in program evaluation. Effectiveness (of programs)—Degree of impact that a program has on identified clients, client problems, and targeted behaviors, feelings, knowledge and opinions. Evaluation—Activities undertaken for the purpose of determining program impact on target client groups and social problems. Evaluation design—In a general sense, the planned activities connected with evaluation, including assessing the evaluation environment, assigning goals and objectives, and choosing measures. Evaluator—The person charged with facilitating the design implementation and reporting of the evaluation. GLOSSARY 85 Focus group—A group of people gathered to discuss a common experience (such as completing a domestic abuse program) in an open-ended manner. Goals—General statements that summarize the ultimate impact that the program is supposed to have on client problems. The general nature of goals usually makes them difficult to measure or fully attain. Mandatory reporting—Reporting to law enforcement or social service authorities that is required by law or by case law of the practitioner. Mean—See average. Measure—A method for systematically collecting information. Median—The middle measure (such as score) in a distribution so that half the measures are above and half below it. In an even number of scores, compute the average of the two middle scores to obtain the median. Mode—Category with the largest number. Outcome objectives—Statements that define and fit within the scope of stated goals. Objectives describe the effects that services are intended to have on the problem. Objectives are realistic and attainable in light of the services provided. Objectives are measurable and are stated in a way that includes a level of achievement (often in terms of increase/decrease by a specific number or percent) within a given time frame. Program—All the activities directed toward a specific goal. Qualitative data—Information of a subjective nature, such as might be collected through in-depth interviews. Quantitative data—Structured information that can be measured in numbers, such as might be collected through questionnaires in which responses are made by checking off a number within a given range to indicate frequency or degree. Results—What you learned about your program from conducting evaluation. Safety plan—A statement of the specific strategies which a battered woman and her children may use when feeling threatened by danger or violence. Standardized questionnaires—Measures developed through a series of psychometric testing and standardized against a large group of people considered to represent the range of possible responses in society. Statistical significance—When applying tests of significance to data, the likelihood of coming to this conclusion has only a small possibility of being in error. Unintended outcomes (or side effects)—Effects that programs may have on clients that are not the result of planning. 86 EVALUATING DOMESTIC VIOLENCE PROGRAMS Appendix A Sample Measures: Part I Therapy Unit Follow-up Forms 88 EVALUATING DOMESTIC VIOLENCE PROGRAMS Release Form to Contact Partner To: (current partner, ex-partner or person you were abusive to) Name Address Phone It is the policy of the Domestic Abuse Project to require that you give DAP written permission to contact your partner, ex-partner, etc. We require this for the following reasons: 1. We make every attempt to call your partner, introduce ourselves and our programs (Women’s Program, Children’s and Adolescent Program) and extend an invitation to your partner to sign up for these services. 2. We call your partner to briefly explain the Men’s Program and encourage your partner to call if they have any questions and concerns. 3. We might call your partner if you leave the group angry or escalated and we have reason to fear for your partner’s safety. 4. We will contact you and your partner approximately six months after you leave the program to help us determine our program effectiveness. 5. DAP has a strict policy regarding confidentiality, and we do not discuss with your partner the content of your work in group or details of what you discuss. We share information with your partner only if we have doubts about your progress in the program or concerns for your partner’s safety. I authorize the Domestic Abuse Project to disclose to the above named person(s) any information pertinent to my participation at the Domestic Abuse Project. Client signature Date Facilitator signature Date APPENDIX A — SAMPLE MEASURES I 89 90 EVALUATING DOMESTIC VIOLENCE PROGRAMS Release of Information To: Re: I authorize you to release the following information from my treatment records to the Domestic Abuse Project (indicated with a YES or NO in the space provided for each of the following): Narrative account of case history, diagnosis, progress, and recommendations for future counseling programs Psychological testing results Phone contact Other I also authorize the Domestic Abuse Project to disclose to the above named person any information pertinent to participation in my treatment program. I understand this disclosure will be used for the purpose of receiving therapeutic services at the Domestic Abuse Project. This will expire one year from the date signed unless revoked sooner by myself. Client signature Date Facilitator signature Date Requested by on Sent by on APPENDIX A — SAMPLE MEASURES I 91 92 EVALUATING DOMESTIC VIOLENCE PROGRAMS Women’s Participation Agreement Oppression is a form of violence. Racism, sexism, and any prejudices are oppressive. Racist and sexist remarks are abusive, encourage further violence, and are contrary to the principles of DAP and therefore will NOT be tolerated. Domestic Abuse Project agrees: 11. To provide you with a copy of the Client Rights Statement and to respect your dignity and confidentiality, as defined by that statement. 12. To be honest with you all in all aspects of your program. 13. To provide you with group therapy, individual therapy, couple’s therapy and other therapy when necessary, as goals are met. 14. To provide you with referrals/recommendations in response to additional needs DAP is unable to help you meet. What DAP expects from you I agree: 11. To work on the goals we have agreed upon. 12. To be honest and direct about myself. 13. To attend groups following intake. The number of group sessions varies depending on the type of group. A maximum of three absences will be allowed for all groups. Two absences in a row will not be allowed in the first four weeks of group. In the ongoing group, one make-up session is required. More absences would mean reassessment of my commitment and goals with my therapist. 14. To be on time for group and individual sessions. 15. To call DAP and leave a message at 874 - 7063 if I am unable to attend a group or individual session. If I miss a session without giving any notice, I may be charged for that session. 16. To participate in these groups. This includes sharing experiences, insights, feelings and completing group activities and homework assignments. 17. To follow through on referrals for evaluation and treatment when deemed appropriate by DAP staff (e.g. chemical abuse, psychological evaluation, etc.). 18. To know my fee, to pay it, and to adjust it with the business office if my financial situation changes. 19. Not to be under the influence of alcohol, or other drugs the day of group. (continued on back) APPENDIX A — SAMPLE MEASURES I 93 Women’s Participation Agreement continued 10. In the event that I am abused or violent toward others, to report the incident to my therapist at the next group meeting. 11. If my partner is not a client at DAP, I will keep confidential the meeting time and place of my group. (This may help to ensure my own safety and the safety of DAP staff and clients.) 12. I agree to be non-violent while participating in the Domestic Abuse Project Women’s Program. This agreement includes being non-violent with children and others, as well as with my partner. (Information and assistance about non-violent discipline is available.) 13. Other. 14. (client’s initials) To help DAP determine program effectiveness, I give my permission to have a DAP staff member contact me six months after I have completed participation in DAP services. I have read and I understand the information provided about clients rights, confidentiality, and exceptions to confidentiality. Client signature Date Intake counselor Date Therapist’s name Group starts 94 EVALUATING DOMESTIC VIOLENCE PROGRAMS Program Goals for Women Personal Goals Group Goals 1. To become aware of and express thoughts and feelings appropriately. 2. To increase my feelings of self-esteem. 3. To develop and use an effective protection plan to learn and believe I am not responsible for the violence. 4. To heal from the effects of violence and know I have choices. Client signature Date Intake therapist Date APPENDIX A — SAMPLE MEASURES I 95 96 EVALUATING DOMESTIC VIOLENCE PROGRAMS INSTRUCTIONS FOR THERAPY UNIT FOLLOW-UP Domestic Abuse Project Revised June 1996 APPENDIX A — SAMPLE MEASURES I 97 Instructions Responsibilities in the Follow-up Process The Domestic Abuse Project’s follow-up procedures involve a variety of agency personnel with differing roles. The six areas of responsibility are: Director of Evaluation and Research Responsible for the design and refinement of follow-up measures, hiring and initial training of follow-up callers, and liaison for technical, research related questions. Director of Therapy Responsible for locating missing files and insuring that clinical staff have completed or obtained from clients all required materials. Callers may rely on him/her for answers to questions regarding therapy programs and referral sources for clients in need of services at follow-up. The Director of Therapy will be responsible for providing and maintaining a small packet of referral sources for callers to use when clients need further assistance. Director of Administration Responsible for overseeing the maintenance of the client database that generates lists of program completers and non-completers as well as answering questions regarding agency functioning. Business Manager Callers may rely on him/her to supply the quarterly lists of program completers and noncompleters. Front Desk Personnel Callers may request production of forms and copying of forms from them, as well as expect them to answer general questions regarding agency function. Follow-up Callers Responsible for completing: 1) as many follow-up interviews as required each quarter; 2) an individual follow-up summary sheet for each client for whom data is available; 3) quarterly summary sheets within two weeks following the end of quarter and; 4) distribution of copies of all completed quarterly summary sheets to both the Executive Director and the Director of Evaluation and Research within two weeks following the end of the quarter. 98 EVALUATING DOMESTIC VIOLENCE PROGRAMS Instructions Preparations for Interviews Ask the Business Manager for a list of names for whom data is to be collected. Ask the Business Manager for a separate print out of female and male completers. Each print out will tell you the date of client completion (left hand column under comp) and whether that client is male or female. (In the left hand column under code—numbers which begin with a 1=male and numbers which begin with a 2=female. The numbers themselves are the client’s agency number.) Callers will then prepare the Tally Sheet (form 1140.f2). Every effort will be made for the same caller to contact both partners if they both require a follow-up interview during the same quarter. (In other words, if during the first quarter both the perpetrator and the victim/survivor complete groups at the same time, then both require a follow-up.) Quarters are three month intervals: 1st Quarter is January/February/March, 2nd Quarter is April/May/June, 3d Quarter is July/August/September, and 4th Quarter is October/November/ December. On the Tally Sheet be sure to group clients according to the quarters they finished. In other words, on each Tally Sheet there should only be one quarter represented. So if you are doing 1st Quarter, only clients who finished during January/February/March should be on the Tally Sheet. The next step is locating client files. Locate an agency client file for each name on your list. These client files can be anywhere: 1) check both the open and closed file drawers on the second floor; 2) ask the therapists and; 3) ask the front desk staff. List each file you have found and store it temporarily in the Research drawer. Post a copy of the list by the files and give a copy to the front desk worker. The list will inform others that you have a file and where they can find it if necessary. Use the Tally Sheet to keep track of the contents and progress of each client file. With the file in front of you, fill out the following all at once (trust us it is easier to do this all at once, in the beginning): (1) the Tally Sheet; (2) the Follow-up Interview Schedule and; (3) the Individual Summary Sheet. The information in the client’s file—the initial first call postcard, the intakes, and the closing—give you all of the information that you need to do so. While filling out the above forms, check the file for the following: • Is there a signed consent form for partners of female and male perpetrators? (Completers of women’s groups for non-abusive women do not need to have a consent form for their partners.) • If there is no consent form for the partner of the perpetrator, you may not call the partner, only the perpetrator. • Are both parts of the intake interview, the single pink form and the multi-paged white form, in the file? • If the file is closed (in the closed file drawers), is there a completed closing form? APPENDIX A — SAMPLE MEASURES I 99 Instructions List any client files you cannot find, as well as missing and incomplete forms within files, and give these lists to the Director of Therapy. Do this ASAP and give a due date of when you will need the completed file returned to you. For the Follow-up Interview Schedule, clients’ telephone numbers and addresses may be found: 1) in their files—look at the Client Final Address and Phone Update (clntupdt.frm); 2) in their partner’s files; 3) by asking the Director of Therapy to contact the client’s most recent therapist to obtain current number; 4) by finding their completed Future Services Forms or; 5) by calling 411 for directory assistance. When going through client’s files, write down their full name, with middle initial, and address, as well as the name and address of their partners (for perpetrators). This will help if you need to call 411. Blank copies of all Follow-up Interview Schedules, Individual Summary, and Tally Sheets, along with client files being used should all be kept in the Research Unit’s file drawer on the second floor. For extra copies of forms, ask the front desk for copying assistance. For large jobs you will need to plan ahead and fill out a copying request form with the front desk staff, agreeing with them upon a date for completion. 100 EVALUATING DOMESTIC VIOLENCE PROGRAMS Instructions Who Is Called? Please be respectful and sensitive with your phone calling. Many partners are not in a support program and answering follow-up questions can be distressing. If a partner or a client does not want to be interviewed, respect their decision and simply note their refusal. Confidentiality is expected from each caller. If you know the person being called, please refer that case to the Therapy Director to call. DAP has committed itself to helping a minimum of 60 men and 60 women achieve positive outcomes (nonviolent men and safe women) each year. In a series of studies, we have found that about 2 out of 3 men who complete the program are not violent at follow-up. Therefore, if we complete interviews with a minimum of 90 male program completers or, as preferred, their partners, we will have no problem meeting our goals for the men’s program. While our data is not so clear concerning the women’s program, interviewing only female completers should be required. In sum, callers should attempt to interview ALL women’s program completers and the partners of men who have completed or, if not the partner, then the male completers themselves. Priority Data: The priority data for both men and women are women’s reports Male Completers: For the men who have completed DAP’s programs, the report of his most recent partner is our priority. For male completers, first try to contact his partner. If you successfully complete the interview with her, then stop and move on to the next name on your list. If she cannot be interviewed, then attempt to interview the man. If you interview his partner it is not necessary to interview him. This preference is based upon research that shows women report twice as much violence as men. Women’s reports are a more conservative estimate of the program’s success. Female Completers: Only try to contact the women who have completed women groups. Be certain that the women you are interviewing have completed women’s group and not aftercare. Aftercare completers often show up on the print-outs as completers, but it is only women’s group completers that we are interviewing. To ascertain whether she is a woman’s group or aftercare completer: • Look at the outside of her file for a f/u (follow-up phone call) and date; if there is one, it means she has already been interviewed and the date on the print-out refers to aftercare or; (2) check the closing form (and sometimes the print out) for an A/C, this means she is an aftercare completer—not a women’s group completer—for this quarter. • Do not interview a woman who has completed aftercare unless she has never been interviewed after completing women’s group and she completed women’s group within the last three months. No men will be interviewed for female completers. APPENDIX A — SAMPLE MEASURES I 101 Instructions INTERVIEWING CLIENTS When calling, be careful not to identify yourself as a DAP employee until you are sure you are talking directly to the person that you plan to interview. Again, this is to ensure the safety and confidentiality of the people involved. If you leave messages, leave only your name and DAP’s phone #, and do not leave DAP’s name. It is okay to unblock the DAP phone to call numbers who will not accept blocked calls. Women’s Program As stated above, attempt to contact and interview only women on the women’s group completer list. Use the Victim/Survivor’s Follow-up Interview Schedule for these interviews. If the woman answers yes to #10, saying their children were in DAP’s children’s program, be sure to complete the Children’s Program Evaluation too. No partners are to be contacted for participants in DAP’s women program. If the woman’s partner also happens to be in the quarter’s summary follow-up sample, let her know you will be asking her questions both about her and about her partner. Sometimes, however, she may not have had any contact with her partner and, if so, you will need to contact her partner directly. Men’s Program As stated earlier, the first priority is to contact the men’s most recent partner. If you are successful in interviewing her, you should use the Abuser’s Follow-up Interview Schedule and ask her only questions one through seven on the schedule form. Again, be sure to fill out the Children’s Program Evaluation if they (man or woman) answer yes to #6 on the Abuser’s Follow-up Interview Schedule. There is no need to call the men if the women have completed the Followup Interview Schedule. In the event that you cannot reach a man’s partner, then it is time to try contacting and interviewing the man. Again, use the Abuser’s Follow-up Interview Schedule and complete all the questions. Men should be asked all the questions on the Abuser’s Interview Schedule. Keep a running account of your progress on each case by making the appropriate notes on the Tally Sheet. After each Follow-up Interview Schedule is completed, fill out the Individual Summary Sheet. Again, it is easier and simpler if you do this all at once. When talking with former clients, the need may arise for you to refer them for additional assistance. You should have the following resources when calling: • A list of men’s and women’s self-help groups around the Twin Cities. This is available from the Director of Therapy. • A list of emergency shelter and crisis line numbers including both DAP’s daytime First Call hours and the night-time services available. This is available from the Director of Therapy. 102 EVALUATING DOMESTIC VIOLENCE PROGRAMS Instructions You should always keep these and any other available resources with you when contacting clients. Your interviews should be focused on the Follow-up Interview Schedules, but you should always be sure to provide information and referral when deemed necessary. Again, please recognize that your phone calls can be upsetting for both parties, so be respectful, supportive, and refer to appropriate resources. Near the end of the quarter, when you have contacted every man and woman completer that was possible to contact, a form letter may be sent to the clients you were not able to contact. Before sending the letter, check to make sure the client’s address is correct. Again, check the first call postcard but go by the client’s final address and phone number update. If there is already returned mail in the client’s file, do not send a letter to that same address. If you do not have an address, simply note on your Tally Sheet that you were unable to contact that client. If you send a letter, please be sure that letter is written on plain paper, sent in a plain envelope, and does not identify DAP by name. This is to ensure clients’ confidentiality and, more importantly, safety should others open the letter. Give the list of names and addresses of clients you were not able to contact to the front desk staff along with the form letters’ computer disk number (reschltr.rfm). Let the front desk staff know your name, the days and times to reach you during the week, and the phone number you would like to be called at, so that they may insert that information into the form letter. Note on your request that plain paper stationery and envelope are to be used. APPENDIX A — SAMPLE MEASURES I 103 Instructions INDIVIDUAL SUMMARY SHEETS It is time to complete an Individual Summary Sheet when an interview has been completed or all attempts have been made to contact the client (and, in the case of men, his partner) have failed. You should have your completed Follow-up Interview Schedule when you complete the summary. Individual summaries are completed for all clients regardless of whether or not they were contacted and interviewed. (The Individual Summary Sheet should already be completed for all interviewees.) Individual Summary Sheets are designed so that intake, closing, and follow-up data are in columns from the left (intake) to the right (follow-up). The item number from, which the data is to be drawn, is listed in the parentheses before the blank line in which the data is to be entered. When all Individual Summary Sheets are completed at the end of the quarter, they are divided into two groups: 1) male completers and; 2) female completers. General Note: It is customary to use 9 to note that there is no data available. Write N/A if the question is not applicable (i.e. children living with). 104 EVALUATING DOMESTIC VIOLENCE PROGRAMS Instructions QUARTERLY SUMMARY SHEETS It is important that Quarterly Summary Sheets be completed carefully. There are different Quarterly Summary Sheets for men and women. One should be completed for each of the two groups mentioned above: 1) male completers and; 2) female completers. While general directions follow, go to the actual model sheets for step by step directions. They should be in the follow-up research binder in the research drawer. On the Abuser Quarterly Summary Sheet, the first category listed is Goal IA (2) Threat Summaries. You should tally the number of men for whom women’s reports were collected. Figure how many of those men for whom women’s reports were collected. Determine how many of those men were reported to be not threatening by their female partners. DO NOT YET include data from men’s self-reports or clinician’s estimates. In other words, you will find: 1. MEN NOT threatening at Follow-up (reported by women) = ____ of ____ Then follow these steps: 1. Enter the number of men reported by women to be not threatening in the left blank space. Look on the Individual Summary Sheet: Abuser for this information. If there are no threats reported by the victim under follow-up, then the men are assumed to be non-threatening. 2. Enter the total number of female partners interviewed in the right blank space. 3. Put aside the Individual Summary Sheets used in steps 1 and 2 above. That is, put aside for the moment the data collected from men’s partners. Just below the line you just completed on the Quarterly Summary Sheet is a line that reads: 2. MEN NOT threatening at Follow-up (self-report) = ______ of _______ Now follow these steps: 1. Enter the number of men self-reported to be not threatening in the left blank space. Again, look on the Individual Summary Sheet: Abuser for this information. If there are no threats reported by the abuser under follow-up, then the men are assumed to be non-threatening 2. Enter the total number of men interviewed in the right blank space. 3. Data entered on this line of the Quarterly Summary Sheet should not include data from cases included on the first line. 4. Now also put aside the Individual Summary Sheets used in steps 1 and 2 for this second line. That is, put aside for the moment the data collected from men’s selfreports as well as women’s reports. APPENDIX A — SAMPLE MEASURES I 105 Instructions The next line should read: 3. MEN NOT threatening by clinician’s estimate at closing = _____ of _____. This line will include data from only those cases where no follow-up interviewers could be conducted but where clinician completed Closing Forms. Follow these steps: 1. Enter the number of men reported by CLINICIANS to be not threatening in the left blank space. 2. Enter the total number of men upon which CLINICIAN data was the only available data in the right blank space. 3. Data entered on this line of the Quarterly Summary Sheet should not include data from cases included on the first and second line. Finally you will find: 4. No data _____ of ______ Enter in the number of men for whom there was no data available. These are men for whom there were no women’s interviews, no men’s interviews, and no clinician closing forms. Add the numbers entered in the left columns of items 1, 2, and 3: you have the total number of men not threatening. Add the numbers entered in the right columns of items 1, 2, and 3: You have the number of men for whom data was available. If you add this to the number entered in item 4, you would have the total number of men on your original calling list. In short, the item numbers 1-4 are rank ordered in terms or the most important data. Women’s reports are most important. If we don’t have these we look for men’s self-reports and, if not these, then the clinician estimates. If you have done this correctly, data for cases entered on one line should not be entered on the other lines. Each zip code, age, and level of education should be listed and followed by the number of people in the sample in that zip code, age, or level of education. There is also space for those clients who had no data for each of those categories. The rest of the demographic listings should be selfexplanatory. 106 EVALUATING DOMESTIC VIOLENCE PROGRAMS Instructions Post Follow-up Responsibilities A copy of the Quarterly Summary Sheets along with all the Individual Summary Sheets are kept in the Research Unit’s file drawer. A copy of the Quarterly Summary Sheets is given to the Executive Director and another to the Director of Evaluation and Research. The last page of the Follow-up Interview Schedule is an evaluation form that is torn off and given to the Director of Therapy. The Children’s Program Evaluations should also be given to the Director of Therapy. Client files can be returned to their proper places and crossed off of the check-out list. Before filing, mark the cover of the files with F/U and the date of the follow-up. If the client was not contacted, then write no contact under the data on the folder. APPENDIX A — SAMPLE MEASURES I 107 Instructions Finally Congratulations on completing a most complicated task! Your continued effort and accuracy may some day help us better understand and prevent domestic violence. 108 EVALUATING DOMESTIC VIOLENCE PROGRAMS Quarter_____________Year____________ Client Name Completion Summary Form Therapist Partner’s Name Intake Date Completion Date Consent Closed File Located Follow-up Completed Mailed Kids APPENDIX A — SAMPLE MEASURES I 109 110 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part I Date _________________ Age _________________ Home phone ________________ Work phone_________________ Home phone ________________ Work phone_________________ Name Spouse/Partner’s name 1. Is this the person to whom you have been violent? ___ Yes ___ No 2. Will your partner be attending DAP? ___ Yes ___ No 3. If you are separated from your partner, was separation due to violence? ___ Yes ___ No 4. How did you find out about our program? Enter the # of the answer here: 1. 2. 3. 4. 5. Self Spouse/partner Family member DAP client 5. 6. 7. 8. Hotline Shelter Media Other agency 9. Child protection 10. Courts 11. Other, specify What is your ethnic background? Enter the # of the answer here: _______ 1. 2. 3. 4. 5. 6. African American/Black American Indian Caucasian/White Chicano/Hispanic/Latino Asian American Other, specify APPENDIX A — SAMPLE MEASURES I 111 Abuser Intake Form: Part I 6. What was your approximate income last year? Enter the # of the answer here: _______ 1. 3. 5. 7. 7. 8. Less than $5,000 $10,000 to $14,999 $20,000 to $24,999 $30,000 or more 2. $5,000 to $9,999 4. $15,000 to $19,999 6. $25,000 to $29,999 What is the highest level of education that you have finished? Enter the # of the answer here: ______ Elementary school 1 2 3 4 5 6 College/vocational school 13 14 15 16 Junior high 7 8 9 Post-college 17 18 19 20 Senior high 10 11 12 Does your partner currently have an Order For Protection (OFP) or a Restraining Order? ___ Yes ___ No 9. Have charges ever been pressed against you for violation of an Order for Protection? ___ Yes ___ No 10. Has your partner ever pressed charges against you for assault? ___ Yes ___ No Describe the assault: Result: 112 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part I 11. Have the police ever pressed charges against you for assault? ___ Yes ___ No 12. Were you referred or ordered to DAP by the courts? ___ Yes ___ No 13. Have you had an intake at DAP before? ___ Yes ___ No 14. Do you think your current alcohol/drug use is excessive? ___ Yes ___ No 15. Does your chemical use have a negative impact on your work or home life? ___ Yes ___ No 16. Have you been through chemical dependency treatment in the last 6 months? ___ Yes ___ No 17. Are you currently under a doctor’s care for depression or anxiety? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 113 Abuser Intake Form: Part I 18. Are you currently taking any medication for this? ___ Yes ___ No If yes, what kind of medication are you taking? 19. Have you had serious and persistent thoughts regarding suicide or hurting yourself or others in any way? ___ Yes ___ No 20. Have you had mental health services and/or psychiatric hospitalization? ___ Yes ___ No Please identify: 114 Therapist’s name Agency Dates of service Name of hospital Date of hospitalization Reason EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part II Date Name Client # DOB Intake facilitator Note: Check to make sure all blanks are filled in. 1. When was the last incident of physical abuse toward your partner? Date Describe: 2. When was the last incident of any other kind of abuse toward your partner? Date Describe: 3. I am going to go through a list of behaviors that may have occurred when you and your spouse/partner had a dispute, or at any other time. I would like you to try to remember how frequently each behavior occurred during the last six months you were together. (Place # of the answer on the blank line next to each question.) 1 Not at all 2 Rarely 3 Occasionally 4 Frequently 5 All the time How frequently did you: 11. Discuss issues relatively calmly ......................................................................___ 1 2. Express feelings using words like “I feel sad” or “I feel hurt”........................___ 13. Gather more information instead of jumping to conclusions ..........................___ 14. Try to find a compromise solution ...................................................................___ 15. Listen to your partner ......................................................................................___ 16. Take your partner’s opinion into account ........................................................___ 17. Leave the room to calm down when you felt yourself getting really upset...........................................................................................___ 18. Stomp out in the middle of an argument .........................................................___ APPENDIX A — SAMPLE MEASURES I 115 Abuser Intake Form: Part II 1 1 Not at all 2 Rarely 3 Occasionally 4 Frequently 5 All the time How frequently did you: 19. Scream at or insult your partner ......................................................................___ 10. Belittle your partner .........................................................................................___ 11. Sulk or withdraw affection ..............................................................................___ 12. Interrupt your partner’s eating or sleeping to bother your partner ..................___ 13. Say your partner couldn’t leave or spend time with certain people ................___ 14. Verbally pressure your partner to have sex ......................................................___ 15. Physically harm pets ........................................................................................___ 16. Physically discipline children ..........................................................................___ 17. Make threats to leave your partner, harm children or take them away, have an affair, or withhold money ....................................___ 18. Threaten to hit or throw something at your partner .........................................___ 19. Throw, hit, kick, or smash objects ...................................................................___ 20. Drive recklessly to frighten your partner .........................................................___ 21. Burn your partner.............................................................................................___ 22. Push, grab, or shove your partner ....................................................................___ 23. Slap or spank your partner with an open hand ................................................___ 24. Bite or scratch your partner .............................................................................___ 25. Hit your partner with something ......................................................................___ 26. Physically force your partner to do something she/he didn’t want to do ........___ 27. Physically force your partner to have sex ........................................................___ 28. Punch your partner with your fist ....................................................................___ 29. Kick your partner .............................................................................................___ 30. Direct blows to your partner’s stomach when she was pregnant ....................___ 31. Throw your partner bodily ............................................................................... 32. Beat your partner unconscious ........................................................................___ 33. Choke or strangle your partner ........................................................................___ 34. Threaten your partner with a knife, gun, or other weapon ..............................___ 35. Use a weapon against your partner ..................................................................___ 36. Anything that hasn’t been mentioned? ............................................................___ Specify: 116 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part II 4. Describe your most violent incident toward your current partner. Date: __________ 5. Has your partner ever received medical treatment as a result of the violence? ___ Yes ___ No Specify: 6. Has your partner needed medical treatment but didn’t get any? ___ Yes ___ No 7. Have you been abusive to a partner in any past relationships? ___ Yes ___ No Describe: 8. As an adult, have you been physically violent against . . .? a) current partner ___Yes ___No b) other partners ___Yes ___No c) children ___Yes ___No d) other family members ___Yes ___No e) friends ___Yes ___No f) other (specify) ___Yes ___No 9. Do you and your partner have disagreements about sex? ___ Yes Describe: ___ No APPENDIX A — SAMPLE MEASURES I 117 Abuser Intake Form: Part II 10. Do you pressure your partner to be sexual when she/he doesn’t want to? ___ Yes ___ No If no, how do you handle this? Family of Origin 11. Who was in your family when you were growing up? 12. What circumstances affected your family life (chemical abuse, violence, poverty)? 13. Was physical violence part of your parent(s)’s/guardian(s)’s relationship? ___ Yes ___ No Between whom? Please describe. 14. Were you physically abused by anyone in your family? ___ Yes ___ No By whom? Please describe. (Objects or weapons used) 118 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part II 15. Were you verbally or emotionally abused in your family? ___ Yes ___ No 16. Were you sexually mistreated while growing up? ___ Yes ___ No 17. Have you ever felt suicidal? ___ Yes ___ No If yes, explore when and under what circumstances. 18. Are you currently feeling suicidal? ___ Yes ___ No Describe: Children 19. Do you have children? ___ Yes ___ No Name Age Lives with: APPENDIX A — SAMPLE MEASURES I 119 Abuser Intake Form: Part II 20. Have the children witnessed or heard the violence? ___ Yes ___ No Describe their reaction: 21. How do you discipline? (NA if not applicable) ___ ___ ___ ___ Talk and explain rules Scold Yell Put in corner ___ ___ ___ ___ Restrict to house Restrict friends Restrict phone Time out 22. Do you use physical punishment (bruises or marks)? ___ Yes ___ No ___ Slapping ___ Spanking ___ Hit with objects ___ Shaking ___ Punching ___ Kicking 23. Does your partner use physical punishment? ___ Unknown ___ Yes ___ No ___ Slapping ___ Spanking ___ Hit with objects 120 EVALUATING DOMESTIC VIOLENCE PROGRAMS ___ Shaking ___ Punching ___ Kicking Abuser Intake Form: Part II 24. Is/has Child Protection ever been involved with your family? ___ Yes ___ No CPS worker County Phone 25. Comments regarding Child Protection involvement: Chemical Use History 26. Are you currently taking medication? ___ Yes ___ No What kind and for what reason? 27. How often do you use alcohol or other drugs? 28. What kind? 29. Have you been through chemical dependency treatment? ___ Yes ___ No Where? When? Completed? APPENDIX A — SAMPLE MEASURES I 121 Abuser Intake Form: Part II 29a. Are you currently in a 12-step program? ___ Yes ___ No Where? How often do you attend? Do you have a sponsor? 29b. How long have you been sober/drug free? 30. Have you had legal consequences as a result of your use? ___ Yes ___ No Describe: 31. How did/does your use affect your work life? 32. How did/does your use affect your home life? Legal History 33. Have you ever been arrested? ___ Yes ___ No 122 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Intake Form: Part II Results: 34. Have you ever served time in jail, workhouse, or prison? ___ Yes ___ No Facilitator’s Impressions Rate client attitude: 1 Completely blames partner 2 3 Knows violence is wrong but is unconvinced he needs a program 4 5 Takes full responsibility for his behavior Intake Facilitator: If you have any of the following concerns, circle “yes” and bring to team meeting to determine acceptability. 1. Multiple problems — too numerous or beyond staff expertise Yes No 2. Child sexual abuse or incest perpetrator Yes No 3. Client terminated from another program Yes No 4. Primary form of abuse has been sexual Yes No 5. Referred by Child Protection Services Yes No 6. Generalized violence Yes No 7. No physical abuse reported Yes No 8. Extreme resistance — may be disruptive to group Yes No 9. Recent sobriety may indicate need for a “no use” contract Yes No 10. Other Yes No APPENDIX A — SAMPLE MEASURES I 123 Abuser Intake Form: Part II Disposition Accepted into program Referred out To: Rejected Why: Intake facilitator 124 EVALUATING DOMESTIC VIOLENCE PROGRAMS Date Closing Summary Form: Client Working to Be Non-Abusive Client name DOB Partner’s name Client number Date # Sessions 1. Orientation Intake Completion of education Completion of group Date of last in-person contact Date of closing Facilitator/therapist primary therapist Person closing case 2. Where was client in the treatment sequence at time of last contact? (Circle one.) 1. Education 2. Intake 3. Men’s Group 4. Individual 5. Other 3. Reason for closing case. (Circle one.) 1. 2. 3. 4. 5. 6. Terminated from Education a. Too many absences b. Other No show for a. Intake b. 1st Process Group c. Other Clinician termination a. Too many absences from Process Group b. Other Referred to more appropriate resource/agency Completed Other, please specify: _____________________________________________________ 4. What services did the client receive? Individual counseling Other ___________________ ___ Yes ___ No ___ Yes ___ No On completion client referred to : Aftercare Parenting Group Couples/Family ___ Yes ___ No ___ Yes ___ No ___ Yes ___ No Partner involvement: Was partner involved at DAP? ___ Yes ___ No APPENDIX A Number of sessions ______ ______ ______ ______ ______ — SAMPLE MEASURES I 125 Closing Summary Form 5. Was the client ever referred out for services other than DAP? CD assessment Psychological assessment Individual counseling Family counseling Group counseling Other, please specify: ________________________ ________________________ ___ ___ ___ ___ ___ Yes Yes Yes Yes Yes ___ ___ ___ ___ ___ No No No No No ___ Yes ___ No ___ Yes ___ No 6. Since the time of intake, how many violent incidents have been reported at the following levels of severity? (Write in the number of incidents.) Toward partner Toward children Toward other Threats of violence (restricted physical movement, intentionally interrupted sleeping or eating, threatened to hit, threw or hit something, drove recklessly, etc.) Violence (threw something at other, punched, restrained, or wrestled, slapped, choked or strangled, physically forced sex, punched, burned, beat unconscious, threatened with or used weapon, etc.) 7. Since the time the abuser entered group, have any threats or actual physical violence toward children by the abuser been reported? (Use definitions above.) ___ Yes ___ No 8. Clinician estimate of success in program Scale: 1 poor 2 fair 3 average 4 good 5 excellent 9 N/A A. Ability to be non-violent at this time. _____ B. Ability to avoid using threats of violence at this time. _____ 126 EVALUATING DOMESTIC VIOLENCE PROGRAMS Closing Summary Form C. Ability to recognize cues to violence and takes steps to de-escalate. _____ D. Ability to use time-out appropriately. _____ E. Improvement in attitudes that help end violence: e.g., accept responsibility for violence, decrease in sexist attitudes. _____ F. Ability to recognize and stop destructive self-talk at this time. _____ G. Level of group participation: e.g., extent of participation, providing and receiving feedback, taking risks, etc. _____ H. Ability to use self-control plan. _____ I. Use of group members for support outside of group. _____ J. Ability to empathize with partner’s point of view. _____ 9. Social isolation scale: Clinician’s estimate of social isolation. Enter the # of the answer here: 1. Lack of contact with people outside immediate family. 2. Some contact with people outside family; some emotional support. 3. Good support system of people outside family for emotional support. APPENDIX A — SAMPLE MEASURES I 127 128 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Follow-up Interview Schedule Client name Client # Address Phone H: W: Partner’s name Is partner a client? ___ Yes ___ No Phone H: W: Interviewer contact notes Date Time Remarks Date Time Remarks If an interview was completed, whom was it with? ___ Abuser ___ Victim/survivor If interview was not completed, what was the reason? Abuser Victim/survivor Interviewer Date APPENDIX A — SAMPLE MEASURES I 129 Abuser Follow-up Interview Schedule Script: Hello, my name is , representing the Evaluation Unit of the Domestic Abuse Project. To Victim/Survivor: What we’re doing is trying to evaluate the program your partner went through, and what I want to do, IF it’s OK with you and IF you have the time, is to ask you a few questions about your impressions of the program and how you’re doing now. I would like to ask you some questions about your partner’s behavior since he(she) left the Domestic Abuse Project. The information you give me will be confidential (except in cases where reporting is mandated by the law), and this interview is completely voluntary. To Abuser: What we’re doing is trying to evaluate the program you went through, and what I want to do, IF it’s OK with you and IF you have the time, is to ask you a few questions about your experiences in the program and how you’re doing now. I would like to ask you some questions about your behavior since you left the Domestic Abuse Project. The information you give me will confidential (except in cases where reporting is mandated by the law), and this interview is completely voluntary. To the Interviewer: Interviewee gave verbal permission to interview. ___ Yes ___ No 1. To Abuser: What is your relationship now to the partner that you abused? To Victim/Survivor: What is your relationship now to the partner who was abusive? (Please check one.) ___ 1. ___ 2. ___ 3. ___ 4 ___ 5. ___ 6. ___ 7. ___ 8. 2. Dating Living with partner (not married) Married Separated Divorced Widowed No Relationship Other Have you had any contact with your partner since you (or s/he) left DAP? (Contact includes in person, by phone, and by mail.) ___ Yes If Yes, what type and how often? __________________________ ___ No If No for victim/survivor, go to #21. If No for abuser, answer this: Do you have a new partner since leaving DAP? 130 ___ Yes If yes, continue. ___ No If no, go to #7. EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Follow-up Interview Schedule 3. Since leaving DAP, have you (has your partner) been physically violent against: a) current partner b) other partners c) children d) other family members e) friends f) other, specify ___ ___ ___ ___ ___ ___ Yes Yes Yes Yes Yes Yes ___ ___ ___ ___ ___ ___ No No No No No No If yes, remember #6. 4. Now I’m going to go through a list of behaviors that may have occurred with you and your partner. I would like you to try to remember how frequently each behavior occurred — and this is only since you (your partner) left DAP. The choices are: 1 Not at all 2 Rarely 3 Occasionally 4 Frequently 5 All the time How frequently did you (your partner): 1. Discuss issues relatively calmly 2. Express feelings using words like “I feel sad” or I feel hurt” 3. Gather more information instead of jumping to conclusions 4. Try to find a compromise solution 5. Listen to your partner (you) 6. Take your partner’s (your) opinion into account 7. Leave the room to calm down when you (he/she) felt yourself (him/herself) getting really upset 8. Stomp out in the middle of argument 9. Scream at or insult your partner (you) 10. Belittle your partner (you) 11. Sulk or withdraw affection 12. Interrupt your partner’s (your) eating or sleeping to bother your partner (you) APPENDIX A — SAMPLE MEASURES I 131 Abuser Follow-up Interview Schedule 13. Say your partner (you) couldn’t leave or spend time with certain people 14. Verbally pressure your partner (you) to have sex 15. Physically harm pets 16. Physically discipline children 17. Make threats to leave your partner (you), harm children or take them away, have an affair, or withhold money 1 Not at all 2 Rarely 3 Occasionally 4 Frequently How frequently did you (your partner): 18. Threaten to hit or throw something at your partner (you) 19. Throw, hit, kick or smash objects 20. Drive recklessly to frighten your partner (you) 21. Burn your partner (you) 22. Push, grab, or shove your partner (you) 23. Slap or spank your partner (you) with an open hand 24. Bite or scratch your partner (you) 25. Hit your partner (you) with something 26. Physically force your partner (you) to do something she/he (you) didn’t want to do 27. Physically force your partner (you) to have sex 28. Punch your partner (you) with your (his/her) fist 29. Kick your partner (you) 132 EVALUATING DOMESTIC VIOLENCE PROGRAMS 5 All the time Abuser Follow-up Interview Schedule 30. Direct blows to your partner’s (your) stomach when she was (you were) pregnant 31. Throw your partner (you) bodily 32. Beat your partner (you) unconscious 33. Choke or strangle your partner (you) 34. Threaten your partner (you) with a knife, gun, or other weapon 35. Use a weapon against your partner (you) 36. Anything that hasn’t been mentioned? Specify: 5. If any violence has been reported: Where you (was your partner) using alcohol or other chemicals during any violent incidents that have occurred since DAP? ___ Yes ___ No For victims/surviors, go to back of last page #21 for comments. 6. If abuse to children is reported in #3: Has this abuse been reported to Child Protection? ___ Yes ___ No If no, notify interviewee of reporting requirement as mandated reporter if there is injury to the child-bruises, welts, burns, fractures, swellings. Nature of injury: ABUSER SECTION ONLY 7. Have you used your control plan since leaving DAP? ___ Yes ___ No ___ No need APPENDIX A — SAMPLE MEASURES I 133 Abuser Follow-up Interview Schedule 8. At present, how likely is it that you can identify the cues leading to a violent incident? (Please circle answer below.) 1 Not likely 9. 3 Moderately likely 2 Slightly likely 3 Moderately likely 5 Extremely likely 4 Strongly likely 5 Extremely likely How would you compare the amount of emotional support you get from people outside your immediate family now as compared to when you started coming to DAP? (Place a check mark by the answer.) ___ Significantly less emotional support ___ About the same ___ Significantly more 11. 4 Strongly likely If cues occurred, how likely is it that you would use your control plan? (Please circle answer below.) 1 Not likely 10. 2 Slightly likely ___ Somewhat less ___ Somewhat more Are you receiving, or have received, counseling or chemical dependency treatment since leaving DAP? ___ Yes ___ No Client Follow-up Feedback To Therapist Section Therapist name Now I have some separate questions about what you thought about the group. At the end of this part, I’ll ask you if you’d like me to put your name on this part or not. We give your counselor the information on these last questions to help him/her know what has been helpful or not helpful. We can give him/her this information anonymously or with your name. If you do not want your counselor to see this at all, we will honor that request as well. (Counselors do not see the answers to any of the other questions — those are kept confidential for research only). 12. Do you feel that you have changed since you came DAP? ___ Yes ___ No 134 EVALUATING DOMESTIC VIOLENCE PROGRAMS Abuser Follow-up Interview Schedule How would you say that you have changed since you started the program? 13. What would you identify as the most helpful experience during the program in helping you to control your violence? 14. What would you identify as the most helpful experience in helping you to accept responsibility for your violent behavior? 15. What would you identify as the least helpful experience during the program in helping you to control you violence? 16. Do you have any suggestions as to how the program could be improved? 17. All things considered, how would you rate the Domestic Abuse Project? (Please circle answer below.) 1 Poor 2 Fair 3 Good 4 Excellent 18. In general, how are you doing now? 19. If not currently an active case: Do you feel in need of support or counseling now? ___ Yes If yes, provide information on aftercare groups. ___ No APPENDIX A — SAMPLE MEASURES I 135 Abuser Follow-up Interview Schedule 20. Do you have any other comments you would like to make to those in charge of Domestic Abuse Project? That’s the last question. Would you like one to put your name on this last part, or just leave it off? ___ Yes Name _______________________________________ ___ No Thank you very much for your time and cooperation. 21. VICTIMS/SURVIVORS: Do you have any comments you’d like to make about the men’s program or about DAP? They can be made anonymously. That’s the last question. Would you like one to put your name on this last part, or just leave it off? ___ Yes Name ___________________________________ ___ No Thank you very much for your time and cooperation. 136 EVALUATING DOMESTIC VIOLENCE PROGRAMS Individual Summary Sheet: Abuser Client name________________________________________________Client #____________ Date of Follow-up interviewer name__________________________________summary_____________ ______________________________________________________________________________ 1. THREATS Intake Closing Follow-up Victim Abuser 8B =________ None 4(11-15) Indirect threats: Y N 4(16-19) Direct violence: Y N YN YN (success = 3,4, or 5) (success = both N in either column) ________________________________________________________________________ 2. VIOLENCE Intake Follow-up Victim Abuser None 8A =________ 4(20-25) Direct violence: Y N YN 4(26-34) Severe violence: Y N YN (success =3,4, or 5) (success = both N in either column) and a score of 1 for 11- 34 _______________________________________________________________________________ 3. CHILD ABUSE Intake (#9) Violence to children Y N (#24) C.P. services Y N Children living with Y N Child Abuse Summaries: Intake: YES NO Closing Closing 7 Violence to children Y N Follow-up 3(C) Violence to children Y N 4 (15) Physical discipline Y N Follow-up: YES NO APPENDIX A — SAMPLE MEASURES I 137 Individual Summary Sheet: Abuser 4. ATTITUDES Intake None Closing Follow-up 8E=______improved attitudes None 8F=______recognize destructive self-talk (success = 3, 4, or 5 on both) _______________________________________________________________________________ 5. USE OF CONTROL PLAN Intake Closing Follow-up None 8C = _______cues Used self-control plan? Y N NN 8D = _______time-out Likely identify cues_________ _______________________________________________________________________________ 6. SOCIAL ISOLATION Intake Closing Follow-up None 10=_________ 10=_________ (success = 2 or 3) (success = 3,4, or 5) _______________________________________________________________________________ 7. DEMOGRAPHICS Intake Closing Follow-up Zip code__________ 3 = Y N Completer None Age______ 4 = _______ Total of sessions Ethnic code________ Approximate income___________ Highest education_________ Court ordered? ___ Y ___ N 138 EVALUATING DOMESTIC VIOLENCE PROGRAMS Quarterly Summary: Abuser N = _________ Quarter of follow-up _________ to ________ Missing files = ____________ Quarter (circle one): 1 2 3 4 Year_______ _______________________________________________________________________________ GOAL IA (2) Threat summaries 1) MEN NOT threatening at follow-up (reported by women) =_________ of ________ 2) MEN NOT threatening at follow-up (self-report) = ___________ of ___________ 3) MEN NOT threatening by clinician’s estimate at closing = ________ of _________ 4) NO DATA = _____________ of _____________ 5) Total = ______________ of ______________ _______________________________________________________________________________ GOAL IA (1) Violence Summaries 1) MEN NOT violent at follow-up (reported by women) =____________ of __________ 2) MEN NOT violent at follow-up (self-report) = _____________ of _____________ 3) MEN NOT violent by clinician’s estimate at closing = __________ of ____________ 4) NO DATA = ______________ of _______________ 5) TOTAL = ________________ of _______________ _______________________________________________________________________________ GOAL IA Child Abuse Summaries 1) MEN NOT abusing at intake: YES = ____ NO = ____ NO DATA = ___________ 2) MEN NOT abusing at closing: YES = ____ NO = ____ NO DATA = ____________ 3) MEN NOT abusing at follow-up: YES = ____ NO = ____ NO DATA =___________ GOAL IA (3) Attitude Summaries 1) MEN changed attitudes (clinician’s estimate at closing) = __________ of _________ 2) NO DATA = ___________ of ___________ 3) TOTAL = ______________ of _____________ _______________________________________________________________________________ APPENDIX A — SAMPLE MEASURES I 139 Quarterly Summary: Abuser GOAL IA (4) Social Isolation Summaries 1) MEN successfully changed at follow-up = ___________ of ___________ 2) MEN successful by clinician’s estimate at closing = ___________ of ___________ 3) NO DATA = ____________ of ____________ 4) TOTAL _____________ of __________ _______________________________________________________________________________ RESIDENCE (list zip codes) _________( ) __________( ) __________( ) __________( ) __________( ) __________( ) _________( ) __________( ) __________( ) __________( ) __________( ) __________( ) _________( ) __________( ) __________( ) __________( ) __________( ) __________( ) Average age of men = _______ years. Ranging from ______ to ______ years old. No Data = _______ of ________ Number in ethnic code: 1 =______ 2 = ______ 3 = ______ 4 = ______ 5 = ______ 6 =______ 7 = ______ No data = _________ of _________ Average age of education = ______ Ranges from ______ to ______ years. No data = ________ of ________ Number of men in income level: 1 = _____ 2 = _____ 3 = _____ 4 = _____ 5 = _____ 6 = _____ 7 = ______ No data = _______ of _______ Children living with: ________ of _______ No data = ________ of _________ _______________________________________________________________________________ CONTROL PLAN USE 1. 2. 3. 140 Used control plan? YES _______ NO ________ Not Needed = __________ No data = ________ of _________ Likely to identify cues? 1 = _______ 2 = _______ 3 = ________ 4 = ________ 5 = ________ No data = ________ of _________ Likely to use this plan? 1 = _______ 2 = _______ 3 = ________ 4 = ________ 5 = ________ No data = ________ of __________ EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake: Part I Date ___________________ DOB __________________ Work phone _____________ Name _______________________________ Home phone _________________________ Partner or Ex-partner’s name _____________________ 1. 2. 3. 4. 5. 6. 7. Is this the person who was violent with you? ___ Yes ___ No Will this person be attending this program? ___ Yes ___ No Are you living with this person? ___ Yes ___ No If you are separated, was the separation due to violence? ___ Yes ___ No Have you ever gone to a shelter for battered women? ___ Yes ___ No How many times? ______________ How did you find out about our program? Enter the # of the answer here: _______ 1. Self 5. Shelter 9. DAP Client 2. Hotline 3. Child Protection 4. Spouse/partner 6. Courts 7. Other family member 8. Media 10. Other agency 11.Other (specify) ____________________ 8. What is your ethnic background? Enter the # of the answer here: __________ 1. African-American 4. White 2. Asian 5. Chicano/Latino 3. Native American 6. Other (specify) ________ 9. What was your approximate income last year? Enter the # of the answer here: ______ 1. Less than $5,000 4. $15,000 to $19,999 7. $30,000 or more 2. $5,000 to $9,999 5. $20,000 to $24,999 3. $10,000 to $14,999 6. $25,000 to $29,999 10. What is your current occupation? _________________________________ 11. What is the highest level of education you have finished? Enter the # of the answer here: ___ Elementary School 1 2 3 4 5 6 Junior High School 7 8 9 Senior High School 10 11 12 College/Vocational School Post-College 13 14 15 16 17 18 19 20 12. Do you currently have an order of protection? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 141 Victim/Survivor Intake: Part I 13. Have you or the police ever pressed charges for violation of an Order For Protection? ___ Yes ___ No 14. Were the police ever called to your home? ___ Yes If yes, what action did they take? ___ No 15. Did they arrest your partner? ___ Yes ___ No 16. Did you press charges? ___ Yes ___ No 17. Did the police press charges? ___ Yes ___ No 18. Were you referred to DAP by the courts or Child Protection Services? ___ Yes ___ No If yes, please identify: ______________________________________________ __________________ By whom? Phone ___________________________________________________________________ Conditions of participation 142 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake: Part I 19. Have you ever had an intake at DAP before? ___ Yes ___ No 20. Have you had any counseling and or psychiatric hospitalizations in the past? ___ Yes ___ No If yes, please identify: Name of hospital Dates of psychiatric hospitalizations Reasons ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 21. Have you ever participated in any therapy group or support groups? ___ Yes ___ No Type of group Agency/organization Dates of participation ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ APPENDIX A — SAMPLE MEASURES I 143 144 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake Form: Part II Client name DOB Date INTAKE COUNSELOR 1. Would you tell me why you decided to come to DAP? Why did you choose this time for services? 2. When was the last incident of physical violence with your partner? Approximate date: Describe: 3. What is the status of your relationship? Name Separated/together Length of relationship 4. I am going to go through a list of behaviors that may have occurred when you and your partner had a dispute or at any other time. I would like you to try to remember how frequently each behavior occurred in the past six months. I would like you to tell me if the behavior occurred: 1 - Never 2 - Rarely 3 - Occasionally 4 - Frequently 5 - Very frequently How frequently did your partner: 1. Discuss issues relatively calmly ................................................................……… 2. Express feelings using words like: “I feel sad” “I feel hurt”................................. 3. Gather more information instead of jumping to conclusions.................................. 4. Try to find a compromise solution or take your opinion into account................... APPENDIX A — SAMPLE MEASURES I 145 Victim/Survivor Intake: Part II 5. Listen to you......................................................................................……………... 6. Left the room to calm down when getting really upset..................................…...... 7. Tell me how frequently—that is, how many times a day, week or month—the following behavior has occurred during the last six months. 0 - Never 1 - Once or twice 2 - Several times in 6 mos 3 - Once or twice a month 4 - Several times a month 5 - Once or twice a week 6 - Several times a week 7 - Once or twice a day 8 - more than once/day NA - Not applicable How frequently did your partner: 1. 2. 3. 4. 5. 6. 7. 8. 9. Belittle you or insult you Scream at you or bark orders at you Stomp out in the middle of an argument Pout or withdraw affection Interrupt your eating/sleeping to bother you Say you couldn’t leave or spend time with certain people Physically harm pets Physically discipline children Make psychological threats (e.g. threats to leave you, withhold money, harm the children or take them away, have an affair, start drinking, etc.) 10. Threaten to hit you or to throw something at you 11. Threaten to kill you or to kill/hurt himself/herself 12. Throw, hit, kick or smash objects 13. Drive recklessly to frighten you 14. Verbally pressure you to have sex 15. Physically force you to have sex 16. Physically force you to do something you didn’t want to do 17. Push, grab or shove you 18. Slap or spank you with an open hand 19. Punch you with a closed fist 20. Kick you 21. Hit you with something 22. Hit you in your stomach when you were pregnant 23. Throw you bodily 24. Beat you unconscious 25. Choke or strangle you 26. Burn you 27. Any of the following: pull your hair, spit at you, tickle when you wanted him/her to stop, twist your arms or legs, pinch you, bit or scratch you 146 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake: Part II 28. Threaten you with a knife, gun or other weapon 29. Use a weapon against you 30. Anything that hasn’t been mentioned 8. What do you remember as the most violent incident? 9. Did you ever seek medical services as a result of the violence? ___ Yes ___ No Have you ever needed to get medical attention and not gotten it? ___ Yes ___ No 10. Have you experienced physical violence in any past relationship? ___ Yes ___ No With whom? Please describe. FAMILY OF ORIGIN 11. Who was in your family when you were growing up? (Use family diagram with names and ages.) APPENDIX A — SAMPLE MEASURES I 147 Victim/Survivor Intake: Part II 12. What was it like for you growing up in your family? (Relationship of family members, significant losses, chemical abuse, etc.) 13. Did you observe physical violence in your family? ___ Yes Between whom? ___ No 14. Were you physically abused by anyone in your family? ___ Yes By whom? Please describe. (Were objects or weapons used?) ___ No 15. Were you verbally or emotionally abused in your family? ___ Yes By whom? Please describe. ___ No 16. Were you touched by anyone in your family or others in ways that were sexual or made you feel uncomfortable? By whom? If yes, please describe. ___ Yes ___ No 17. Have you been sexually abused by anyone as an adult? ___ Yes By whom? Please describe. ___ No 148 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake: Part II 18. a. As a child or as an adult, have you ever felt suicidal or contemplated suicide? ___ Yes ___ No b. Are you presently feeling suicidal? ___ Yes ___ No CHILDREN 19. Do you have any children? ___ Yes ___ No Name Age Lives with 20. What is your relationship like with each of the children? 21. Have the children witnessed or heard the violence? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 149 Victim/Survivor Intake: Part II Describe their response: 22. How do you discipline? Put a check mark next to each relevant answer. talk and explain rules scold send to room/put in corner restrict phone/friends restrict to house yell/scream spank/slap punish with objects Other: 23. How frequently do you use physical punishment? (bruises or marks?) 24. Does your partner use physical punishment? ___ Yes ___ No Describe: 25. Has your behavior or your partner’s behavior toward your children ever seemed inappropriate or sexual? ___ Yes ___ No 26. Is/has Child Protection ever been involved with your family? ___ Yes ___ No CPS worker (Release needed?) 150 EVALUATING DOMESTIC VIOLENCE PROGRAMS County Phone Victim/Survivor Intake: Part II CHEMICAL USE 27. How often do you use alcohol or other drugs? What kind? 28. Are you concerned about your use? ___ Yes ___ No 29. Have you ever been through chemical dependency treatment? ___ Yes ___ No Where: When: Completed: Yes No Note to intake workers: Do you need a release of information? 30. Have you had any legal involvement as a consequence of your use? ___ Yes ___ No Please describe: 31. When there has been violence, had you been using chemicals? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 151 Victim/Survivor Intake: Part II What percentage of time: 0 1–25% 26–50% 51–75% 76–99% 100% 32. Are you currently taking any medication? ___ Yes ___ No Type: Dosage: 33. What do you see your strengths to be? To the interviewer: If your partner has participated in the Domestic Abuse Project’s program, we may contact you six months after your partner leaves the program. ___ This was read to the client. Counselor Impressions MOOD/AFFECT/GENERAL IMPRESSIONS: 152 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Intake: Part II Intake Counselor: If you have any of the following concerns, check YES and bring to team meeting to determine acceptability. 1. Multiple problems—too numerous or beyond staff expertise. ___ Yes ___ No 2. Primary form of abuse has been sexual. ___ Yes ___ No 3. Referred by Child Protection Services. ___ Yes ___ No 4. Generalized violence ___ Yes ___ No 5. Abuse seems to be minimized or denied ___ Yes ___ No 6. Extreme resistance—may be disruptive to group ___ Yes ___ No 7. Recent sobriety may indicate need for a no-use contract ___ Yes ___ No 8. Other ___ Yes ___ No Disposition Accepted into program Referred out to: Rejected Why: Conditions of participation Intake counselor_____________________________ Date___________________________ APPENDIX A — SAMPLE MEASURES I 153 154 EVALUATING DOMESTIC VIOLENCE PROGRAMS Closing Summary Form: Victim/Survivor Client name Client number Partner’s name Client number Person closing case (if involved in program) Date of intake Date of completion of group Date of last in-person contact Date of closing Primary therapist Where was client in the treatment sequence at time of last contact? Enter the # of the answer here: 1. 2. Intake Women’s Group 3. 4. Individual Other Reason for termination: 1. 2. 3. No-show Drop out Clinician terminated Reason: 4. 5. 6. Referred to more appropriate resource Completed Other, please specify: What services did the client receive? # of sessions Intake DAP Women’s Group Couples Group Parenting Group Individual Counseling Couples/Family Sessions Completer Group Other ___ ___ ___ ___ ___ ___ ___ ___ Yes Yes Yes Yes Yes Yes Yes Yes ___ ___ ___ ___ ___ ___ ___ ___ No No No No No No No No Partner involvement: Was partner involved at DAP? ___ Yes ___ No Was the client ever referred out for services other than DAP? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 155 Closing Summary Form: Victim/Survivor CD Assessment Counseling Individual Counseling Family Counseling Group Treatment Other, please specify: ___ ___ ___ ___ ___ Yes Yes Yes Yes Yes ___ ___ ___ ___ ___ No No No No No Since the time of intake how many violent incidents by this client’s partner have been reported at the following levels of severity? (Write in the number of incidents.) PARTNER CHILDREN OTHER INDIRECT THREATS OF VIOLENCE (restricted physical movement, intentionally interrupted sleeping or eating, etc.) DIRECT THREATS OF VIOLENCE (threatened to hit, threw or hit something, drove recklessly, etc.) DIRECT VIOLENCE (threw something at other, pushed, restrained or wrestled, slapped, etc.) SEVERE VIOLENCE (choked or strangled, physically forced sex, punched, burned, beat unconscious, threatened with or used weapon, etc.) Clinician estimate of success in program: Scale: A. B. C. D. E. F. 156 1 Poor 2 Fair 3 Average 4 Good 5 Excellent 9 N/A Ability to protect self from violence at this time. Improvement in ability to protect self from violence at this time. Improvement in attitudes that lead to self-protection (e.g. disowns responsibility for violence.) Level of group participation (e.g. extent of participation, providing and receiving feedback, taking risks, etc.) Use of group members for support outside of group. Ability to maintain adequate parenting and to nurture. EVALUATING DOMESTIC VIOLENCE PROGRAMS Closing Summary Form: Victim/Survivor Has client gone to a shelter for battered women since intake? ___ Yes ___ No Has client gotten an Order for Protection since intake? ___ Yes ___ No Has client called the police since intake? ___ Yes ___ No Has client pressed charges against partner for violence since intake?___ Yes ___ No Social isolation scale Clinician’s estimate of social isolation. Enter # of the answer here: 1. 2. 3. 4. 5. Lack of contact with other people outside immediate family. Minimal contact with people outside immediate family; minimal emotional support. Some contact with people outside family; some emotional support. Quite a bit of contact outside family for emotional support. Good support system of people outside family for emotional support. APPENDIX A — SAMPLE MEASURES I 157 158 EVALUATING DOMESTIC VIOLENCE PROGRAMS Individual Summary Sheet: Victim/Survivor Client name____________________________________________ Client #____________ Follow-up interviewer name_____________________________ Date of summary____________ Summary______________________________________________________________________ ______________________________________________________________________________ 1. PROTECTION Intake None Closing 8A = _______protect self Follow-up 3 = _____ Used Protection Plan Y N NN 4 = ______identify cues NN 5 =______ likely to use plan NN (success=3, 4, or 5) (success=3, 4, or 5 on both 4 and 5) _______________________________________________________________________________ 2. ATTITUDES Intake None Closing Follow-up 8C = ________ improved attitudes None (success = 3, 4, or 5) _______________________________________________________________________________ 3. SOCIAL ISOLATION Intake Closing Follow-up 10 = _________ 8 = _________ (success = 2 or 3) (success =3,4,5) _______________________________________________________________________________ 4. LEGAL ACTIONS Intake Current OFP? Violation of OFP? YN YN Police called? Charge for assault? YN YN Closing Left home in fear? Received OFP? Called police? Pressed charges? Follow-up Y N Left home in fear? Y N NN Police called? Y N NN Charges for assault? Y N NN Y N Received OFP? Y N NN Y N Violation of OFP? Y N NN Y N Medical treatment? Y N NN APPENDIX A — SAMPLE MEASURES I 159 Individual Summary Sheet: Victim/Survivor 5. PARENTING Intake None Closing Follow-up 8F = _________ recognizes effects None (success = 2, 3, 4, or 5,) Children living with? Y N _______________________________________________________________________________ 6. DEMOGRAPHICS Intake Closing Follow-up Zip code__________ 3 = Y N Completer None Age______ 4 = _______ Total # of sessions Ethnic code________ Approximate income__________ Highest education_________ 160 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Follow-up Interview Schedule Client name _______________________________________ Client # Address Phone H: W: Partner’s name Is partner a client? ___ Yes ___ No Partner’s phone H: W: Interviewer Contact Notes DATE TIME REMARKS DATE TIME REMARKS If an interview was not completed, what was the reason? Interviewer Date APPENDIX A — SAMPLE MEASURES I 161 Victim/Survivor Follow-up Interview Schedule Script: Hello, my name is representing the Evaluation Unit of the Domestic Abuse Project. What we’re doing is trying to evaluate the program you went through, and what I want to do, IF it’s OK with you and IF you have the time, is to ask you a few questions about your impressions of the program and how you’re doing now. I would like to ask you some questions about your situation since you left the Domestic Abuse Project. The information you give me will be confidential (except in cases where reporting is mandated by the law), and this interview is completely voluntary. To the Interviewer: Interviewee gave verbal permission to interview. ___ Yes ___ No 1. What is your relationship now to the partner who was abusive? (Please check one.) ____ 1. ____ 2. ____ 3. ____ 4 ____ 5. ____ 6. ____ 7. ____ 8. 2. Dating Living with partner (not married) Married Separated Divorced Widowed No Relationship Other Have you had any contact with your partner since you left DAP? (Contact includes in person, by phone, by mail.) ___ Yes If yes, what type and how often: ___ No 3. Have you used your protection plan since leaving DAP? ___ Yes ___ No ___ No need 4. At present, how likely is it that you can identify the cues leading to a violent incident? (Please circle one.) 1 Not likely 162 2 Slightly likely 3 Moderately likely EVALUATING DOMESTIC VIOLENCE PROGRAMS 4 Strongly likely 5 Extremely likely Victim/Survivor Follow-up Interview Schedule 5. If cues occurred, how likely is it that you would use your protection plan? (Please circle one.) 1 Not likely 2 Slightly likely 3 Moderately likely 4 Strongly likely 5 Extremely likely 6. Since leaving DAP, have you: a. Left the house and gone somewhere when you were feeling afraid? b. Gone to a shelter or safe house? c. Called the police? d. Press charges against partner for assault? e. Received an order for protection? f. Pressed charges for violation of an order for protection? g. Needed medical treatment because of violence? 7. ___ Yes ___ No ___ No need ___ Yes ___ No ___ No need ___ Yes ___ No ___ No need ___ Yes ___ No ___ No need How would you compare the amount of emotional support you get from people outside your immediate family now as compared to when you started coming to DAP? (Please circle one.) 1 Much less 8. ___ Yes ___ No ___ No need ___ Yes ___ No ___ No need ___ Yes ___ No ___ No need 2 Somewhat less 3 About the same 4 Somewhat more 5 Much more Are you receiving, or have received, counseling or chemical dependency treatment since leaving DAP? ___ Yes ___ No 9. Do you have children living with you in your home? ___ Yes ___ No APPENDIX A — SAMPLE MEASURES I 163 Victim/Survivor Follow-up Interview Schedule 10. 11. Has your child(ren) attended DAP’s Children’s Program? ___ Yes If yes, use child add-on questionnaire now. ___ No If no, continue to #11. Has there been any physical violence against you since you left DAP? ___ Yes ___ No 12. If there are children involved, state: I am going to ask you about any abuse directed toward your children. I am mandated by the law to report this abuse if it has not yet been reported. Since leaving DAP, have you hit, slapped, or otherwise physically abused any children in the family? ___ Yes ___ No If yes: Has this abuse been reported to Child Protection? ___ Yes ___ No Date of report to CPS ____/____/____ Name of worker_______________________ Agency_____________________ If not reported, record injuries here and report to cps: Nature of injury: 164 EVALUATING DOMESTIC VIOLENCE PROGRAMS Victim/Survivor Follow-up Interview Schedule Victim/Survivor’s Follow-up Feedback to Therapist Section Therapist’s name Now I have some separate questions about what you thought about the program. At the end of this part, I’ll ask you if you’d like me to put your name on this part or not. We give your counselor the information on these last questions to help your counselor know what has been helpful or not helpful. We can give your counselor this information anonymously or with your name. If you do not want your counselor to see this at all, we will honor that request as well. (Counselors do not see the answers to any of the other questions—those are kept confidential for research only.) 13. Do you feel that you have changed since you came DAP? ___ Yes ___ No If yes, how would you say that you have changed since you started the program? 14. What would you identify as the most helpful experience during the program in terms of helping you to protect yourself or children from violence? 15. What would you identify as the most helpful experience in helping you to realize that you are not responsible for your partner’s violent behavior? 16. What would you identify as the least helpful experience during the program? 17. Do you have any suggestions as to how the program could be improved? APPENDIX A — SAMPLE MEASURES I 165 Victim/Survivor Follow-up Interview Schedule 18. All things considered, how would you rate the Domestic Abuse Project? (Please circle answer below.) 1 Poor 2 Fair 3 Good 19. In general, how are you doing now? 20. Do you feel in need of support or counseling now? ___ Yes 4 Excellent If yes, provide information on aftercare groups. ___ No 21. Do you have any comments you’d like to make to those in charge of the Domestic Abuse Project? That’s the last question. Would you like one to put your name on this last part, or just leave it off? ___ Yes Name: ___ No THANK YOU VERY MUCH FOR YOUR TIME AND COOPERATION. 166 EVALUATING DOMESTIC VIOLENCE PROGRAMS Children’s Program Evaluation Follow-Up Questionnaire Client # __________ Date of interview ____/____/____ Interviewed by _____________ Read to client: Using the following scale, I’d like to ask how strongly you agree or disagree with the following statements? (Review scale with client.) 1 Strongly disagree 2 Disagree 3 Agree 4 Strongly agree Now, I am going to ask you 5 questions about your child(ren) and ask you to indicate how strongly you agree or disagree with each statement. (Please circle the answer below.) As a result of DAP’s program, my child(ren): SD D A SA Can acknowledge that violence is an issue in their family. 1 2 3 4 Can acknowledge that violence in their family is not their fault. 1 2 3 4 3. Feels better about him or herself. 1 2 3 4 4. Has learned to better protect him or herself. 1 2 3 4 5. Has learned non-violent ways to solve problems. 1 2 3 4 1. 2. APPENDIX A — SAMPLE MEASURES I 167 (Return to adult interview form.) 168 EVALUATING DOMESTIC VIOLENCE PROGRAMS (Data for this form comes from previous forms) Quarterly Summary : Victim/Survivor N = _________ Quarter of follow-up _________ to __________ Missing files = __________ Quarter (circle one) 1 2 3 4 Year__________ _______________________________________________________________________________ GOAL IIA (1) Protection Summaries 1. WOMEN report at follow-up can protect self = _________ of _________ 2. WOMEN can protect self , clinician’s estimate = __________ of __________ 3. NO DATA = _____________ of ____________ 4. TOTAL = _____________ of _____________ _______________________________________________________________________________ GOAL IIA (2) Attitude Summaries 1. WOMEN improved attitude, clinician’s estimate at closing = _________ of ________ 2. NO DATA = __________ of ____________ 3. TOTAL = ____________ of ___________ _______________________________________________________________________________ GOAL IIA (3) Social Isolation Summaries 1. WOMEN successfully changed at follow-up = ________ of ________ 2. WOMEN successfully by clinician’s estimate at closing = ___________ of ________ 3. NO DATA ____________ OF ____________ 4. TOTAL _____________ OF _____________ RESIDENCE (list zip codes) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) __________( ) Average age of women = _______ years. Ranging from ______ to ______ years old. No data = _______ of ________ Number in ethnic code: 1 = ______ 2 = ______ 3 = ______ 4 = ______ 5 = ______ 6 = ______ 7 = ______ No data = _________ of _________ Average age of education = ______ Ranges from ______ to ______ years. No data = ________ of ________ APPENDIX A — SAMPLE MEASURES I 169 Quarterly Summary: Victim/Survey Number of women in income level: 1 = _____ 2 = _____ 3 = _____ 4 = _____ 5 = _____ 6 = _____ 7 = ______ No data = _______ of _______ Children living with: ________ of _______ No data = ________ of _________ _______________________________________________________________________________ CONTROL PLAN USE 1. Used control plan? YES_______ NO________ No data = ________ of _________ Not needed = __________ 2. Likely to identify cues? 1 = _______ 2 = _______ 3 = ________ 4 = ________ 5 = ________ No data = ________ of _________ 3. Likely to use this plan? 1 = _______ 2 = _______ 3 = ________ 4 = ________ 5 = ________ No data = ________ of __________ _______________________________________________________________________________ PARENTING INFORMATION 1. Success at closing by clinician’s estimate: ________ of ________ No data = _______ of ________ Not applicable = _________ of ________ 170 EVALUATING DOMESTIC VIOLENCE PROGRAMS Appendix B Sample Measures: Part II Intervention Satisfaction Survey 172 EVALUATING DOMESTIC VIOLENCE PROGRAMS Intervention Satisfaction Survey Protocols • The Satisfaction Survey will be handed to each woman who is completing contact with the Domestic Abuse Project. • Surveys will be anonymous and completely confidential. Only overall results will be reported. • Surveys will be completed by women while at the office and put in a locked box in the front office. The box will have signs on it assuring confidentiality. • A volunteer, intern, or work-study student will manage the organizing and summarizing of returned surveys - it is possible that the same work-study student conducting the Therapy Unit Follow-up could be used for this process. • The Evaluation and Research Unit will train and supervise the volunteer, intern, or work-study student. • Optional Strategies: a) Should the woman name the primary advocate so we can offer feedback to the advocate? Should there be a separate slip with feedback going directly to the advocate? b) Does utilizing the birth dates and zip codes feel like a sneaky way to identify women and influence the honesty of answers on the questionnaire? c) Does the Domestic Abuse Project wish to offer an incentive for return of survey? Financial or other? Survey Format and Content • Survey will be fit entirely on an 11x17 piece of paper folded in half to resemble a pamphlet. • Survey will take no more than 10 minutes to complete. • Survey questions will focus on satisfaction with advocacy services from the Legal Advocacy Unit, as well as a few questions that focus on other parts of the system. Rev: 12/23/96 APPENDIX B — SAMPLE MEASURES II 173 174 EVALUATING DOMESTIC VIOLENCE PROGRAMS Mailing ID #: ________ Intervention Satisfaction Survey for the Domestic Abuse Project (DAP) You may have recently had contact with a court advocate through DAP’s Legal Advocacy Unit. It would help us if you could take a few minutes to answer some questions so we can improve our services. This survey should only take a few minutes to finish. If you have trouble understanding the survey you may call ____________________ at (612)874-4063 for help in filling it out. Your answers are confidential. We only use the number above to know which questionnaires have been returned. Individuals will not be identified, only an overall summary of surveys will be made. APPENDIX B — SAMPLE MEASURES II 175 Service Survey The court advocate you worked with might have been called an “advocate,” a “DAP advocate” or an “advocate from the Domestic Abuse Project.” All of these will be referred to as “a court advocate” throughout this questionnaire. Questions about Court Advocate Services 1. How much do you agree or disagree that court advocates informed you of your legal rights and options? (Circle your answer below.) 1 Strongly disagree 2. 2 3 4 Strongly agree How happy were you with the court advocacy services you received from DAP? (Circle your answer below.) 1 Not happy at all 2 3 4 Very happy 3. What did you like or dislike about the court advocacy services you received? Comment here: 4. Of the advocacy services you used, what was most helpful? (Check your top three choices below.) 176 ___ Receiving a call from an advocate right after the assault or arrest. ___ Information from advocates about my legal rights and choices. ___ An advocate telling me about the court process and information on my case. ___ Having the advocates listen to and support me. ___ Having an advocate with me in court. ___ An advocate giving me information on other services and resources. EVALUATING DOMESTIC VIOLENCE PROGRAMS Service Survey 5. How were advocacy services helpful or not helpful to you? Comment here: Questions about Other Services 6. How happy were you with how the police responded to you? (Circle you answer below.) 1 Not happy at all 7. 2 3 4 Very happy 2 3 4 Very happy How happy were you with the way the case was settled? (Circle your answer below.) 1 Not happy at all 10. 4 Very happy How happy were you with the sentence the judge gave? (Circle your answer below.) 1 Not happy at all 9. 3 How happy were you with how the prosecuting attorney handled the case? (Circle your answer below.) 1 Not happy at all 8. 2 2 3 4 Very happy What more would you like us to know about these other services? Comment here: APPENDIX B — SAMPLE MEASURES II 177 Service Survey Questions About Access to Services 11. What race do you consider yourself? (Check one.) ___ White/Caucasian ___ Mexican/Hispanic/Latino ___ Black/African American ___ American Indian ___ Asian or Pacific Islander ___ Mixed, no dominant racial identification ___ Other, please fill in: ________________ 12. What is your zip code? __ __ __ __ __ 13. What year were you born? 14. What sexual orientation do you consider yourself? __ __ __ __ ___ Heterosexual/straight ___ Gay ___ Lesbian ___ Bisexual ___ Transgender ___ Other, please fill in: ________________ 15. DAP wants to improve services to ALL people. What would make it easier for you to use our services? Comment here: Thank you for taking the time to answer these questions. Please place your completed survey in the locked box in the front office or mail it back to the Domestic Abuse Project. 178 EVALUATING DOMESTIC VIOLENCE PROGRAMS Possible Questions about the Judge and the Court 1. 2. Did your case go to a pretrial hearing? ___ Yes If you answered yes, continue to #2. ___ No If you answered no, stop here. Did your case go to a jury trial? ___ Yes ___ No 3. How strongly do you agree or disagree with the following statement? (Place a check mark next to the item that best reflects your answer.) “I believe the judge’s sentence reflected my opinions.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree 4. How satisfied were you with the court outcome? ___ Very satisfied ___ Satisfied ___ Dissatisfied ___ Very dissatisfied APPENDIX B — SAMPLE MEASURES II 179 180 EVALUATING DOMESTIC VIOLENCE PROGRAMS Possible Order for Protection Questions 1. Have you received, in the past 6 months, an Order for Protection (OFP) excluding your partner from contact with you? ___ Yes ___ No 2. How strongly do you agree or disagree with the following statements? (Place a check next to the item that best reflects your answer.) a) “The OFP was helpful in reducing assaults against me by my partner.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree b) “If I were in a similar situation again, I would seek to obtain an OFP.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree 3. Was the OFP you received ever violated by your partner? ___ Yes If you answered yes, continue to #4. ___ No If you answered no, stop here. APPENDIX B — SAMPLE MEASURES II 181 Possible Order for Protection Questions continued 4. Did you request help from the police to enforce the OFP? ___ Yes ___ No 5. How strongly do you agree or disagree with the following statements? (Place a check mark next to the item that best reflects your answer.) a) “The police were helpful in enforcing the OFP.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree b) “The city attorney was helpful in enforcing the OFP.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree c) “The courts were helpful in enforcing the OFP.” ___ Strongly agree ___ Agree ___ Disagree ___ Strongly disagree 182 EVALUATING DOMESTIC VIOLENCE PROGRAMS Selected READING Quick study Alter, Catherine and Evans, Wayne (1990). Evaluating Your Practice: A Guide to Self-assessment. New York: Springer. Austin, Michael J. and Associates (1982). Evaluating Your Agency’s Programs. Beverly Hills, California: SAGE. Rutman, Leonard and Mowbray, George (1983). Understanding Program Evaluation. Beverly Hills, California: SAGE. General reference Bloom, M., Fischer, J. and Orme, J. G. (1995). Evaluating Practice (2nd Ed.). Boston: Allyn and Bacon. Denzin, N. K. & Lincoln, Y. S. (Eds.) (1994). Handbook of Qualitative Research. Thousand Oaks, CA: SAGE. Guba, E. G. and Lincoln, Y. S, (1989). Fourth Generation Evaluation. Thousand Oaks, CA: SAGE. Posavac, Emil J. and Carey, Raymond G. (1992). Program Evaluation: Methods and Case Studies (4th edition). Englewood Cliffs, New Jersey: Prentice-Hall. Rossi, Peter H. and Freeman, Howard E. (1993). Evaluation: A Systematic Approach (5th edition). Newbury Park, CA: SAGE. Patton, Michael Quinn (1986). Utilization-focused Evaluation (2nd edition). Beverly Hills, California: SAGE. Patton, Michael Quinn (1987). Creative Evaluation (2nd edition). Newbury Park, California: SAGE. Patton, Michael Quinn (1990). Qualitative Evaluation and Research Methods (2nd edition). Newbury Park, California: SAGE. Sandefur, Gary D., Freeman, Howard E., and Rossi, Peter H. (1986). Workbook for Evaluation: A Systematic Approach (3rd edition). Beverly Hills, California: SAGE. SELECTED READING 183 Domestic violence evaluation Edleson, J. L. and Syers, M. (1990). The Relative Effects of Group Treatments for Men Who Batter. Social Work Research & Abstracts, vol. 26, pp.10-17. Edleson, J. L. and Syers, M. (1991). The Effects of Group Treatment for Men Who Batter: An 18-month Follow-up Study. Research in Social Work Practice, vol.1, pp. 227-243. Edleson, J. L. (1996). Controversy and Change in Batterers’ Programs. In J. L. Edleson and Z. C. Eisikovits (Eds.). Future Interventions with Battered Women and Their Families: Visions for Policy, Practice and Research, pp. 154-169. Thousand Oaks, CA: SAGE. Gondolf, E. W., Yllo, K. and Campbell, J. (1995). Collaboration Between Researchers and Advocates. Paper presented at the Fourth International Family Violence Research Conference, Durham, New Hampshire, July 21-25. (http://www.umn.edu/mincava and look under Education Resources/Scholarly Papers) Peled, E. and Edleson, J. L. (1995). Process and Outcome in Small Groups for Children of Battered Women. In E. Peled, P. G. Jaffe, and J. L. Edleson (Eds.). Ending the Cycle of Violence: Community Responses to Children of Battered Women, pp. 77-96. Newbury Park, CA: SAGE. Syers, M. and Edleson, J. L. (1992). The Combined Effects of Coordinated Intervention in Woman Abuse. Journal of Interpersonal Violence, vol. 7, pp. 490-502. Tolman, R. M. and Edleson, J. L. (1995). Intervention for Men Who Batter: A Review of Research. In S. Stith and M. A. Straus (eds.). Partner Violence: Prevalence, Causes, Consequences and Solutions, pp. 262-273. Minneapolis, MN: National Council on Family Relations. (http:// www.umn.edu/mincava and look under Papers and Reports). Measurement Bloom, M., Fischer, J. & Orme, J. G. (1995). Evaluating Practice: Guidelines for the Accountable Professional (Second Edition). Boston: Allyn & Bacon. Edleson, Jeffrey L. (1985). Rapid-assessment Instruments for Evaluating Practice with Children and Youth. Journal of Social Service Research, vol. 8, pp. 17-31. 184 EVALUATING DOMESTIC VIOLENCE PROGRAMS Filsinger, Erik E. (Ed.) (1983). Marriage and Family Assessment: A Sourcebook for Family Therapy. Beverly Hills, CA: Sage. Fischer, J. & Corcoran, K. (Eds.) (1994). Measures for Clinical Practice: A Sourcebook/Volume 1: Couple, Families & Children. New York: Free Press. Fischer, J. & Corcoran, K. (Eds.) (1994). Measures for Clinical Practice: A Sourcebook/Volumne 2: Adults. New York: Free Press. Fredman, Norman and Sherman, Robert (Eds.) (1987). Handbook of Measurements for Marriage and Family Therapy. Frey, James H. (1989). Survey Research by Telephone (2nd ed.). Newbury Park, CA: Sage. Hudson, Walter W. (1982). The Clinical Measurement Package: A Field Manual. Homewood, Illinois: Dorsey Press. (Also available in computerized package) Levitt, John L. and Reid, William (1981). Rapid-assessment Instruments for Practice. Social Work Research and Abstracts, vol.17, pp.13-19. Sweetland, Richard C. and Keyser, Daniel J. (General Eds.) (1991). Tests: A Comprehensive Reference for Assessments in Psychology, Education, and Business. (3rd ed.) Webb, Eugene J., Campbell, Donald T., Schwartz, Richard D. and Sechrest, Lee (1966). Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally. Sample measures for domestic violence program evaluation Gondolf, E. W. (1991). A Victim-based Assessment of Court-mandated Counseling for Batterers. Criminal Justice Review, vol. 16, pp. 214-226. Henderson, S., Duncan-Jones, P., Byrne, D. G. & Scott, R. (1980). Measuring Social Relationships: The Interview Schedule for Social Interaction. Psychological Medicine, vol. 10, pp. 723-734. Hudson, W. W. (1982). The Clinical Measurement Package. Homewood, IL: Dorsey Press. Hudson, W. W. & McIntosh, S. R. (1981). The Assessment of Spouse Abuse: Two Quantifiable Dimensions. Journal of Marriage and the Family, pp. 873-888. SELECTED READING 185 Lewis, B. Y. (1985). The Wife Abuse Inventory: A Screening Device for the Identification of Abused Women. Social Work, vol. 30, pp. 32-35. Marshall, L. L. (1992). Development of the Severity of Violence Against Women Scales. Journal of Family Violence, vol. 7, pp. 103-121. Mitchell, R. E. & Hodson, C. A. (1983). Coping with Domestic Violence: Social Support and Psychological Health Among Battered Women. American Journal of Community Psychology, vol. 11, pp. 629-654. Saunders, D. G. (1991). Procedures for Adjusting Self-reports of Violence for Social Desirability Bias. Journal of Interpersonal Violence, vol. 6, pp. 336-344. Saunders, D. G., Lynch, A. B., Grayson, M. & Linz, D. (1987). The Inventory of Beliefs About Wife Beating: The Construction and Initial Validation of a Measure of Beliefs and Attitudes. Violence & Victims, vol. 2, pp. 39-57. Shepard, M. F. & Campbell, J. A. (1992). The Abusive Behavior Inventory: A Measure of Psychological and Physical Abuse. Journal of Interpersonal Violence, vol. 7, pp. 291-305. Straus, M. A. & Gelles, R. J. (Eds.) (1990). Physical Violence in American Families. New Brunswick, NJ: Transaction Publishers. Tolman, R. M. (1989). The Development of a Measure of Psychological Maltreatment of Women by Their Male Partners. Violence & Victims, vol. 4, pp.159-177. Special topics Krueger, Richard A. (1988). Focus Groups: A Practical Guide for Applied Research. Newbury Park, California: SAGE. Smith, M. F. (1989). Evaluability Assessment: A Practical Approach. Boston, Mass: Kluwer Academic Publishers. Yin, R. K. (1989). Case Study Research: Design and Methods. Newbury Park, CA: SAGE. 186 EVALUATING DOMESTIC VIOLENCE PROGRAMS We hope the information in this manual has been helpful to you. If you have any suggestions for future editions, please contact us at: Domestic Abuse Project 204 West Franklin Ave. Minneapolis, MN 55404 Tel: (612) 874-7063 Fax: (612) 874-8445 E-mail: dap@mndap.org