Psychotherapy

2015, Vol. 52, No. 2, 180 184

2014 American Psychological Association

0033-3204/15/$12.00 http://dx.doi.org/10.1037/a0037659

Using Client Outcome Monitoring as a Tool for Supervision

Joshua K. Swift

Jennifer L. Callahan

University of Alaska Anchorage

University of North Texas

Tony G. Rousmaniere, Jason L. Whipple,

and Kyle Dexter

University of North Texas

Elizabeth R. Wrape

This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

University of Alaska Fairbanks

A large body of research has supported the use of client outcome monitoring and client feedback in
psychotherapy. However, discussions between supervisors and trainee clinicians in supervision are still
largely based on subjective appraisals made by the trainees. In this article, we discuss 3 strategies for
integrating client outcome data and feedback into the supervisory process: training students to obtain and
use objective client feedback, using specific client data to inform discussions of clients, and identifying
patterns of outcomes across clients to facilitate supervisee growth and development.
Keywords: supervision, outcome tracking, client outcomes, training, practicum

monitoring system (e.g., Outcome Questionnaire system; OQ;

Lambert, Kahler, Harmon, Burlingame, & Shimokawa, 2011; Partners for Change Outcome Management System; PCOMS; Miller
& Duncan, 2004) on a session-by-session basis, comparisons of
individual client change to expected empirically or theoretically
derived patterns of change, and feedback to the providing clinician
regarding whether the client is making adequate progress.
The idea of outcome monitoring was first formally introduced as
patient-focused research by Howard and colleagues in the 1990s
(Howard, Moras, Brill, Martinovich, & Lutz, 1996). Howard et al.
indicated that although clinicians should be concerned with
whether a particular intervention works under both experimental
and naturalistic conditions, they should be most concerned with
whether it is working for the particular client. However, research
has indicated that even experienced clinicians have difficulty
knowing if a treatment is working for a particular client based just
on their own clinical judgment (Lambert, 2010). For example, in
one study Hannan and colleagues (2005) tested the accuracy of
clinical judgment for 48 different clinicians by asking them to
indicate the progress of their clients. Although the clinicians predicted deterioration for only three of the 550 clients, 40 of them
had actually gotten worse during therapy (only one of which was
in the group of three that was predicted). In contrast, the clinicians
predicted just over 500 of their clients to have a positive outcome,
but only a little over 200 actually evidenced progress. A significant
body of research has demonstrated that clinical prediction can be
improved, and in turn better outcomes obtained, via outcome
monitoring systems. In a meta-analysis of two outcome systems
(OQ and PCOM), Lambert and Shimokawa (2011) found that
when outcome monitoring and feedback were included in treatment, clients were three times more likely to make a reliable

Although tracking client change through norm-referenced standardized measures has become more common among clinicians;
supervisees reports of the treatment process and outcome to their
supervisors continues to primarily consist of sharing subjective
appraisals (ODonovan, Halford, & Walters, 2011). In contrast, we
present several methods for use of client outcome monitoring as a
tool for use within supervision. Following a brief review of the
theory and research behind outcome monitoring, we present three
ways to more fully incorporate client outcome monitoring in
supervision: (1) training students to obtain and use objective client
feedback, (2) using outcome monitoring to inform discussions of
specific clients in supervision, and (3) using patterns of outcomes
across clients to facilitate supervisee growth and development. A
brief description and a supervisory vignette are provided for each
skill.1

Outcome MonitoringTheory and Research

Outcome monitoring is defined as the process of routinely
assessing client progress through objective measures to not just
observe a clients degree of change, but to inform clinician decision making based on the individual pattern of progress or lack
thereof. It typically involves the use of a standardized outcome

This article was published Online First October 6, 2014.

Joshua K. Swift, Department of Psychology, University of Alaska Anchorage; Jennifer L. Callahan, Department of Psychology, University of
North Texas; Tony G. Rousmaniere, Student Health and Counseling Center, University of Alaska Fairbanks; Jason L. Whipple and Kyle Dexter,
Department of Psychology, University of Alaska Fairbanks; Elizabeth R.
Wrape, Department of Psychology, University of North Texas.
Correspondence concerning this article should be addressed to Joshua K.
Swift, Department of Psychology, University of Alaska Anchorage, 3211
Providence Drive SSB303, Anchorage, AK 99508. E-mail: Joshua
.Keith.Swift@gmail.com

1
To protect supervisor identities and maintain confidentiality, the vignettes included in this paper are amalgamations of several supervisory
cases and all names have been changed.

180

This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

OUTCOME MONITORING AND SUPERVISION

change and half as likely to end therapy in a deteriorated state

compared to clients in the no feedback conditions.
Given the value of outcome monitoring and obtaining patient
feedback, some have suggested that it be used more fully in
supervision (Lambert & Hawkins, 2001; ODonovan et al., 2011;
Sparks, Kisler, Adams, & Blumen, 2011; Worthen & Lambert,
2007). The existing literature has suggested several benefits associated with outcome informed supervision. First, and perhaps
foremost, it allows supervisory discussions of clients to be based
on client-reported clinical outcomes rather than the supervisees
reporting of his or her perceptions of client progress. When this
process occurs in supervision it sets an example for trainees to
follow in their later clinical work. Second, referring to specific
client data can provide a road map to identify clients it might be
most appropriate to discuss in a given supervisory session. Third,
it can help supervisors step back and see the big picture with a
trainee clinicians work, thus being able to target patterns that
appear across clients with their supervisees.

Training Students in Obtaining Client Feedback

Many of the basics of training in outcome monitoring can
actually happen outside of supervision. For example, in the classroom students can be taught about the rationale behind patientfocused research, they can read the research that supports the use
of client feedback systems, and they can seek to gain an understanding of the meaning of clinically significant and reliable
change. In addition, in an orientation to the specific clinic setting
students can be taught about when, where, and how to administer
the measures, how to use the site specific technology for the
system/measures, and how to access software generated reports (if
available).
However, the more complex aspects of obtaining and making
the most of client feedback (e.g., assisting clinicians in developing
an outcome monitoring plan) are better suited to occur during
supervision. Given the complexity of these tasks, we recommend
that supervisors personally learn about the process of outcome
monitoring and also seek to integrate outcome monitoring into
their own practice before adopting an outcome informed supervision approach (see Whipple & Lambert, 2011). Informed supervision may start by helping trainees select among available outcome
monitoring systems, discussing issues of reliability and validity
(e.g., focusing on accuracy in predicting treatment failure, sensitivity to change), ease of use, quality of the feedback that is given,
cost, and other considerations. Although clinicians may have little
choice about which system to use while in training depending on
the policies of the specific clinic, students who have discussed how
to think about these issues with their supervisors will be better
prepared to evaluate future options and make decisions.
Supervisors also can augment the general outcome system/
measures that are administered in a set way for all clients by
helping trainees learn how to add measures that are suited to
particular clients and/or theoretical orientations. As just a few
examples, supervision may include discussion of adding a disorder
specific symptom rating scale (for depression, anxiety, pain, etc.),
or measures of hope, motivation, social support, or the therapeutic
alliance. Given the unique pieces of information that many different measures could provide, it takes a certain level of skill to
decide which measures are most appropriate while not taking up

181

too much of the clients time. Helping trainees learn to strike this
balance may be a concerted long-term goal of informed supervision. Among the therapeutic variables that could be assessed, a
measure of the therapeutic alliance may be a particularly good
place to start with this line of training, given the value that this
variable plays in both treatment outcomes (Horvath, Del Re,
Fluckiger, & Symonds, 2011) and premature termination (Sharf,
Primavera, & Diener, 2010). One caveat worth keeping in mind
and discussing during informed supervision is that client ratings of
alliance tend to be negatively skewed (Horvath et al., 2011). As
such, even slightly less than ideal ratings may be indicative of
problems in the therapeutic alliance.
The task of discussing the results of outcome monitoring with
clients includes many nuances and is perhaps a more advanced
skill to acquire during informed supervision. However, work toward competency in this area should still begin early in training.
Novice, prepracticum trainees can be taught how to word the
results of the measures and the change patterns in a manner that is
easily understood by clients even before they begin seeing clients.
In contrast, more advanced practicum or internship trainees are
more likely to benefit from supervision focused on how to discuss
feedback in a way that improves understanding of the clients
experience, leads to a specific therapeutic plan or modifications to
the plan, increases the sense of collaboration between the client
and therapist, and facilitates client insight into his or her problems
and experience of change or lack thereof. In addition to overt
instruction, we suggest that informed supervision also encompass
good supervisor modeling. Just as trainee clinicians are asked to
administer appropriate measures to their clients, supervisors may
want to utilize appropriate measures of supervision outcomes (e.g.,
Bahrick, Russell, & Salmi, 1991; Efstation, Patton, & Kardash,
1990; Rousmaniere & Ellis, 2013) and model discussion of those
results as illustrated in the following supervisory vignette.
Clinician: [While sharing a graph of a clients outcome
scores plotted over sessions with the supervisor]
The client that we talked about last week didnt
seem to be doing any better this week and he
ended up rating our relationship lower than
usual.
Supervisor: What did you think when you saw the lower
ratings?
Clinician: I was really kind of surprisedI had been thinking our relationship was fine.
Supervisor: That is the real beauty of outcome monitoring.
With the biases that we have, as clinicians we can
often miss some of the clients feelings and
thoughts about how things are actually going in
psychotherapy. Now that youve seen the clients
lower rating of the therapeutic relationship, what
do you think you can do next?
Clinician: I am not really sure, I feel like we get along well
and he says he likes me.
Supervisor: Sounds like you feel like there is a pretty good
bond between you and the client, but the ratings
were still low. [Examines the working alliance

SWIFT ET AL.

182

measure more closely.] If we look at the scores on

the working alliance measure the items dealing
with collaboration are all rated lower than the
items pertaining to the bond. So what might be
some ways to increase the clients sense of
collaboration?

This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Clinician: I guess I can talk to him about it, but I am not

really sure how?
Supervisor: Do you remember a couple weeks back when you
completed that supervisory alliance measure and
you rated our bond as being slightly lower than
the other areas? Do you remember how I brought
that result up to you?
Clinician: Yeah, I do. You thanked me for providing the
feedback on the measure, mentioned that you
noticed the lower score in that area, and then
asked me how I had been feeling about the bond.
After we talked about it some you asked for ideas
about what we might be able to do differently to
strengthen our bond.
Supervisor: How did you feel that conversation went?
Clinician: Really wellEven though we were talking about
the bond, I have also felt a stronger sense of
collaboration in supervision ever sense.
Supervisor: So how would you feel about taking a similar
approach with your client?
Clinician: Makes a lot of sense now, I definitely think I can
do that.

Using Specific Client Feedback to Guide Supervision

Effective supervision depends, in part, on trainee clinicians
identifying what they need help with during supervision. As a
result, it is important for supervisors to engender in their trainees
a habit of integrating quantitative evidence with their budding
clinical intuition to determine how to prioritize clients for discussion within supervision (Worthen & Lambert, 2007). By using
outcome data to prioritize clients for discussion in supervision,
therapeutic rupture, premature termination, or the advent of a crisis
(i.e., hospitalization or self-harm) may be curbed (Lambert, 2010).
By encouraging consistent standardized outcome monitoring and
reviewing that data within supervision, an atmosphere that fosters
trainees critical thinking (an essential competency; Kaslow, 2004)
can be created. It also can provide a mechanism for determining
whether a trainee is using supervision effectively and yield valuable information about whether a trainee is on track in their
developing skills pertaining to integration of subjective and objective clinical appraisals of client change over time. An exchange
illustrating a process of incorporating client outcome measurement
into a challenging supervision interaction (i.e., needing to give
corrective feedback to a somewhat resistant trainee who confuses
a strong alliance with effective treatment) may underscore this
point:

Clinician: Sarah mentioned in our last session that her mom

doesnt think she is progressing as fast as she
should in therapy.
Supervisor: Hmmm, okay. What do you make of that?
Clinician: Im not sure. I mean Sarah is really easy to
engage and I feel like we have talked about some
pretty important stuff. She always tells me how
grateful she is at the end of sessions and how
much she appreciates me.
Supervisor: But something about her mothers comment has
given you pause . . . at least; it felt important
enough for you to bring it into supervision to
discuss.
Clinician: I think that maybe I dont like that her mom was
kind of devaluing the work we have been doing
together when I think it has been going really
well.
Supervisor: That would linger with me as well. I think its
great that you recognized the potential salience of
moms comment. As I think about it though, it
makes me wonder if bringing moms comment
into session might have been a way for Sarah to
gently share with you her own concern about
progress.
Clinician: Hmm [looks uncomfortable].
Supervisor: Im really glad you brought this up; it actually
helps me make sense of something I have been
noticing in your progress notes and want to talk
about with you. You consistently describe a lot of
progress in your narrative description of each
session, even though we have been noticing and
discussing that Sarahs outcome scores have not
reliably changed from session one. Your appraisals of Sarah and Sarahs actual report are not
converging over time.
Clinician: Well, we have only been working together a few
months [sounds dismissive].
Supervisor: True, there is still quite a bit of work to do . . . but,
I would expect to see some early gains by now
and a lessening in her distress. [Discussion of the
data and client change trajectories ensues to inform clinical decision making.]
The practice of monitoring client progress also can be seen as an
ongoing effort to evaluate trainee progress (Dohrenbusch & Lipka,
2006). This comes with the caveat that progress tracking should
not be punitive or grade dependent. Rather, as shown in the
supervision vignette above, it can be used as another piece in the
evaluative puzzle and a learning experience. Trainees that respond
well in cases of client deterioration can gain confidence, while
those who do not can learn from their missteps within the process
of supervision. This is, of course, dependent on the atmosphere
that the supervisor creates. While supervisory working alliance is
conceptualized as an interaction between both trainee and super-

OUTCOME MONITORING AND SUPERVISION

visor, it is ultimately the supervisor who is responsible for maintenance of a nontoxic learning environment.

This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Using Patterns of Client Feedback to

Guide Supervision
Another method for using clinical outcomes in supervision is
teaching trainees how to examine their overall aggregate outcomes; to review the big picture of their clinical work. As shown
in Figure 1, trainees may periodically be asked to create a spreadsheet that lists all of their clients. For each case, trainees put their
clients basic demographic data (age, gender, race), starting outcome score, final outcome score, number of sessions, primary
presenting problem or clinical focus, and the trainees subjective
opinion about the major challenges and successes of each case.
The supervisor and trainee then review the spreadsheet together,
with an eye for trends in the trainees work and identification of
trainees clinical strengths and weaknesses, which can help counter
the tendency toward self-idealization or self-devaluation that may
be commonly experienced by some trainees (SalzbergerWittenberg, Henry, & Osborne, 1983). For example, a trainee
might show a trend of positive gains with anxiety cases, but less
effectiveness with depression cases. In this case, the supervisor
could shift their training focus toward teaching techniques to
address depression. In another example, a trainee may have a
preponderance of dropouts with a certain demographic group, such
as older women. The supervisor could help the trainee reflect on
what may be limiting their effectiveness with these clients. In
another example, a trainee might show initial positive gains with
many clients, but thereafter a lack of progress. In this case, the
supervisor could discuss the phase model of change (Howard,
Lueger, Maling, & Martinovich, 1993), and how to help clients
move from remoralization to remediation and rehabilitation.
Another tool to get a big picture look at a trainees work is to use
a spreadsheet function to compute the trainees average outcome
change score or effect size (for a review of different methods for
computing effect sizes, see Seidel, Miller, & Chow, 2014). The

183

trainees average change score or effect size can be compared to

other trainees, presuming they work in similar conditions with
similar client populations (Minami, Brown, McCulloch, & Bolstrom, 2012). Note that this method is not recommended if the
trainee has a small number of clients (e.g., fewer than 10 30
clients, depending on the measure). The following is a transcript of
a supervision session that demonstrates how the spreadsheet can be
used in supervision.
Supervisor: Looking at these results overall, what trends
stand out to you?
Clinician: I did notice that all three of my African American
clients got better.
Supervisor: Right. It might be worth reflecting on what
worked with these clients, and helped them stick
around for more sessions. Do you notice any
other trends? What about presenting problems?
Clinician: I seem to do better with anxiety and worse with
depression.
Supervisor: I noticed that too. It looks like two, possibly three,
of your four clients with depression may have
dropped out. Do you have any ideas about that?
Clinician: Honestly, I never fully understood the treatment
approach I was using. I felt like I was just reading
out of the manual. The clients might have noticed
that.
Supervisor: Okay, so sounds like we will want to talk more
about what was working with your African American clients and also review the treatment for
depression that you have been learning. Looking
at the spreadsheet, do you notice anything else?

Figure 1. Example spreadsheet of client data for use in supervision to identify a clinicians strengths and
training needs. OQ come Questionnaire 45.2; WAISR Working Alliance InventoryShort Revised.

SWIFT ET AL.

184

This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Summary
Tracking client outcomes and feedback for use in psychotherapy
has been credited with reduced deterioration and improved clinical
outcomes (Lambert, 2010). Considering previously documented
parallels between psychotherapy and supervision, it is logical to
extend this evidence-based practice to guide and focus supervision.
However, to date, supervision usually consists of only sharing
primarily subjective appraisals (ODonovan et al., 2011).
To our knowledge, only one study has examined whether outcome informed supervision impacts client outcomes. In this study,
Reese et al. (2009) assigned 28 trainees and nine supervisors to
feedback and no feedback conditions. Although those therapists in
the outcome informed supervision condition had a mean prepost
therapy effect size of d 0.92 for their clients, those in the
no-feedback condition had a mean effect size of d 0.23. This
study provides preliminary evidence in favor of outcome informed
supervision, but a design limitation obscures clarity. More specific,
therapists in the no-feedback condition with supervisors were not
able to independently review their clients outcome scores either.
It cannot be ruled out that the observed outcome differences
between groups resulted from differential clinician use of outcome
monitoring rather than informed supervision.
We suggest three pathways for greater incorporation of standardized, norm-referenced client outcome data into the process of
supervision, with corresponding vignettes to illustrate each application: (1) training students to obtain and use objective client
feedback, (2) using outcome monitoring to inform discussions of
specific clients in supervision, and (3) using patterns of outcomes
across clients to facilitate supervisee growth and development.
Although support for these techniques can be found in the literature on tracking client outcomes, research is needed to directly
examine whether outcome informed supervision will have a positive impact on the supervisory process.

References
Bahrick, A. S., Russell, R. K., & Salmi, S. W. (1991). The effects of role
induction on trainees perceptions of supervision. Journal of Counseling
& Development, 69, 434 438. doi:10.1002/j.1556-6676.1991.tb01540.x
Dohrenbusch, R., & Lipka, S. (2006). Assessing and predicting supervisors evaluations of psychotherapistsAn empirical study. Counselling
Psychology Quarterly, 19, 395 414. doi:10.1080/09515070601106737
Efstation, J. F., Patton, M. J., & Kardash, C. M. (1990). Measuring the
working alliance in counselor supervision. Journal of Counseling Psychology, 37, 322329. doi:10.1037/0022-0167.37.3.322
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W.,
Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for
identifying clients at risk for treatment failure. Journal of Clinical
Psychology, 61, 155163. doi:10.1002/jclp.20108
Horvath, A. O., Del Re., A. C., Fluckiger, C., & Symonds, D. (2011).
Alliance in individual psychotherapy. Psychotherapy, 48, 9 16. doi:
10.1037/a0022186
Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovich, Z. (1993). A
phase model of psychotherapy outcome: Causal mediation of change.
Journal of Consulting and Clinical Psychology, 61, 678 685. doi:
10.1037/0022-006X.61.4.678

Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996).
Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51, 1059 1064. doi:10.1037/0003-066X
.51.10.1059
Kaslow, N. J. (2004). Competencies in professional psychology. American
Psychologist, 59, 774 781. doi:10.1037/0003-066X.59.8.774
Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, and feedback in clinical practice. Washington, DC:
American Psychological Association.
Lambert, M. J., & Hawkins, E. J. (2001). Using information about patient
progress in supervision: Are outcomes enhanced? Australian Psychologist, 36, 131138. doi:10.1080/00050060108259645
Lambert, M. J., Kahler, M., Harmon, C., Burlingame, G. M., &
Shimokawa, K. (2011). Administration and scoring manual for the
OQ 45.2. Salt Lake City, Utah: OQMeasures.
Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback.
Psychotherapy, 48, 7279. doi:10.1037/a0022238
Miller, S. D., & Duncan, B. L., (2004). The Outcome and Session Rating
Scales: Administration and scoring manual. Chicago, IL: Author.
Minami, T., Brown, G., McCulloch, J., & Bolstrom, B. J. (2012). Benchmarking therapists: Furthering the benchmarking method in its application to clinical practice. Quality & Quantity, 46, 1699 1708. doi:
10.1007/s11135-011-9548-4
ODonovan, A., Halford, W. K., & Walters, B. (2011). Towards best
practice supervision of clinical psychology trainees. Australian Psychologist, 46, 101112. doi:10.1111/j.1742-9544.2011.00033.x
Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A., Halstead,
J. L., Rowlands, S. R., & Chisholm, R. R. (2009). Using client feedback
in psychotherapy training: An analysis of its influence on supervision
and counselor self-efficacy. Training and Education in Professional
Psychology, 3, 157168. doi:10.1037/a0015673
Rousmaniere, T. G., & Ellis, M. V. (2013). Developing the construct and
measure of collaborative clinical supervision: The supervisees perspective. Training and Education in Professional Psychology. Advanced
online publication. doi:10.1037/a0033796
Salzberger-Wittenberg, I., Henry, G., & Osborne, E. (1983). The emotional
experience of learning and teaching. London, England: Routledge &
Kegan Paul. doi:10.4324/9780203393093
Seidel, J. A., Miller, S. D., & Chow, D. L. (2014). Effect size calculations
for the clinician: Methods and comparability. Psychotherapy Research,
24, 470 484. doi:10.1080/10503307.2013.840812
Sharf, J., Primavera, L. H., & Diener, M. J. (2010). Dropout and therapeutic alliance: A meta-analysis of adult individual psychotherapy. Psychotherapy, 47, 637 645. doi:10.1037/a0021175
Sparks, J. A., Kisler, T. S., Adams, J. F., & Blumen, D. G. (2011).
Teaching accountability: Using client feedback to train effective family
therapists. Journal of Marital and Family Therapy, 37, 452 467. doi:
10.1111/j.1752-0606.2011.00224.x
Whipple, J. L., & Lambert, M. J. (2011). Outcome measures for practice.
Annual Review of Clinical Psychology, 7, 87111. doi:10.1146/annurevclinpsy-040510-143938
Worthen, V. E., & Lambert, M. J. (2007). Outcome oriented supervision:
Advantages of adding systematic client tracking to supportive consultations. Counseling and Psychotherapy Research, 7, 48 53. doi:10.1080/
14733140601140873

Received April 2, 2014

Revision received June 26, 2014
Accepted July 7, 2014