Monitoring Quality of Care in Family Planning

Programs: A Comparison of Observations

And Client Exit Interviews
By Ruth E. Bessinger and Jane T. Bertrand

the approach of “let 100 flowers bloom.”

Context: Monitoring quality of care in family planning programs is important, but the complex- Subsequently, the EVALUATION Project
ity and subjectivity of the topic create many challenges. convened a working group of researchers
to study quality of care, and this group re-
Methods: The Quick Investigation of Quality (QIQ) was developed to monitor quality of care by
duced the list to 42 process indicators.4
using observations of client-provider interactions, exit interviews with clients and facility audits.
The QIQ was field-tested in multiple countries in 1998–1999. Using linked data for 583 clients
The successor project, MEASURE Eval-
in Ecuador, 539 in Uganda and 736 in Zimbabwe, this analysis examines the comparability of
uation, developed and field-tested a low-
results from observations and exit interviews. cost, practical approach to monitoring
quality of care, later named the Quick In-
Results: For a given indicator, levels of agreement between data from observations and inter- vestigation of Quality (QIQ).* The project
views varied across countries, but within a country, results were consistent between instruments.
used an approach based on the Delphi
For the three countries combined, agreement was good to excellent (kappas, 0.47–0.98) on 13
method, in which experts share their
of the 14 indicators examined; observations and exit interviews yielded consistent responses in
knowledge through questionnaires and
63–99% of cases. Agreement was highest on the indicators that measured interpersonal rela-
feedback: It asked specialists in quality of
tions. Inconsistencies reflected primarily that clients received information outside of the observed
care, family planning service delivery or
client-provider interaction.
program evaluation to select from a list of
Conclusions: Observations and client exit interviews provide very similar results for many in- some 80 indicators† the 25 that in their
dicators. However, while some programs may opt for one instrument over the other because of opinion most directly affected quality out-
resource constraints, the combination provides a fuller assessment of quality of care. comes in terms of clients’ behavior. To col-
International Family Planning Perspectives, 2001, 27(2):63–70 lect data on these 25 indicators, three in-
struments were developed: a facility audit
with selected questions to the program

D
uring the 1990s, quality of care be- With this increased focus on quality, a manager, observation of client-provider
came a central focus for the inter- parallel interest has arisen in developing interactions and selected clinical proce-
national family planning and re- means of measuring quality, for several dures, and exit interviews with clients de-
productive health community. Over the past reasons. First, client-provider interactions parting from the facility (and previously
decade, work in this area has been guided can be understood as intervening ele- observed).‡5 These instruments were field-
by the Bruce-Jain framework, which outlines ments in a causal chain through which or- tested in four countries (Ecuador, Turkey,
six elements of quality: choice of method, ganized family planning efforts meet or Uganda and Zimbabwe) between Octo-
information to the client, technical compe- generate demand for fertility regulation.2 ber 1998 and March 1999 to determine the
tence, interpersonal relations, mechanisms Learning more about these processes with feasibility of data collection and the reli-
to encourage continuity and constellation the aim of improving them can have im- ability of the data. A detailed description
of services.1 Organizations have adopted portant programmatic payoffs. Second, of the development and implementation
variations on this theme, such as the Inter- many programs have undertaken activi- of the QIQ methodology and full results
national Planned Parenthood Federation’s ties to improve quality of care in their fa- from the field test are reported elsewhere.6
(IPPF’s) Client Bill of Rights, later amend- cilities. Without measurement tools, it is With the conduct of facility-based sur-
ed to the Client and Provider Bill of Rights. impossible to know whether these activ- veys on the rise, the field-test data provide
ities have achieved their objectives. Third, an excellent opportunity to address a
*The QIQ was developed in collaboration with the Mon- by investing resources in measuring qual- question of critical importance: Do ob-
itoring and Evaluation Subcommittee of the Maximizing
ity, management sends a message to staff servations of client-provider interactions
Access and Quality (MAQ) initiative, funded by the U.S.
Agency for International Development (USAID). that “quality is important.” As such, mea- and exit interviews with clients yield con-
surement reinforces initiatives to improve sistent results? This question is particu-
†The list of indicators was compiled from several sources,
including a checklist developed by MAQ, the IPPF Client
quality. larly pertinent in the context of QIQ for
Bill of Rights, IPPF’s list of indicators on “Quality of Care The challenge in measuring quality is two reasons. First, consistency in results
from the Gender Perspective” and the EVALUATION the complexity and subjectivity of the
Project’s Handbook of Indicators for Family Planning. topic. Although the Bruce-Jain framework Ruth E. Bessinger is evaluation analyst, ORC Macro In-
ternational, Calverton, MD, USA; Jane T. Bertrand is pro-
‡While other instruments were considered during the outlines six elements of quality, literally fessor and chair, Department of International Health, Tu-
development of the QIQ, only these three were selected, hundreds of possible “subelements” lane University School of Public Health and Tropical
to make the data collection as simple and low-cost as pos- might be measured. A task force created Medicine, New Orleans, LA, USA. Funding for this study
sible. For example, while provider surveys have yield- to explore the measurement of quality in was provided by the U.S. Agency for International De-
ed valuable information in the context of situation analy- velopment (USAID) through the MEASURE Evaluation
1990 identified more than 200 indicators
sis (source: reference 5), they were not essential in the project at the University of North Carolina at Chapel Hill,
current context; moreover, researchers opted for mea- of quality in family planning services.3 cooperative agreement HRN-A-00-97-00018-00. The
suring what providers actually do in counseling and clin- This group recommended experimenta- views expressed in this article are those of the authors
ical procedures, rather than what they say that they do. tion with these different indicators, using and do not necessarily reflect the views of USAID.

Volume 27, Number 2, June 2001 63

Monitoring Quality of Care in Family Planning Programs: Observations and Client Exit Interviews

between the two instruments would lend vorable comments could negatively affect The types of facilities included in the
credibility to the QIQ package of instru- the services they will receive in the future.9 study differed across the three countries,
ments. Second, if it is sufficient to admin- In addition, clients may have such low ex- but these differences should not affect the
ister only one of the instruments, rather pectations of services that even when the research question. In Ecuador, the sample
than both, the cost implications for future quality of services is poor, it exceeds their consisted of all 43 family planning facili-
facility-based surveys could be important. expectations and they report positively on ties run by two nongovernmental orga-
It should be stressed that the two in- their experience. A further concern is re- nizations: Asociación Pro-Bienestar de la
struments differ in terms of the type of in- call bias, which occurs when a respondent Familia Ecuatoriana (APROFE) and Cen-
formation they are best suited to capture. cannot accurately recount what happened tro Médico de Orientación y Planificación
Observation is useful in measuring the ac- during the session. Familiar (CEMOPLAF). The Uganda
curacy and thoroughness of information study used a probability sample of 72
imparted during counseling and in as- Methodology public facilities located in 10 districts
sessing providers’ technical competence Assessment Instruments receiving support from the Delivery of
(which clients are generally not able to do). In this article, we assess two of the three Improved Services for Health (DISH)
By contrast, the exit interview is the only QIQ instruments: observation of client- project and in three districts not receiving
instrument used in QIQ that taps clients’ provider interactions and exit interviews support from that initiative. In Zimbab-
perspectives on the services they have re- with clients. For the first instrument, a we, all 39 facilities receiving support from
ceived. However, information from the trained observer (who usually wore a the Family Planning Service Expansion
two can be compared when clients report white coat to blend into the service deliv- and Technical Support Project (SEATS)
on providers’ actions during counseling ery environment) obtained consent from were surveyed.*
and clinical examinations. the provider and the client to be present
Both methods have limitations. The re- during individual counseling and clinical Analysis
liability of data from observation can be an examination. She used an observation Unique identifying information recorded
issue because observers may interpret the guide with a structured checklist of items on the observation and exit interview
same set of provider actions differently.7 related to quality of care (e.g., whether the forms for each client was used to link the
Observation also introduces the potential provider asked particular questions, data from the two instruments for each
bias that service providers will perform whether specific points of information woman; linked data from the three coun-
better than they might under usual condi- were covered and whether certain clini- tries were combined for the purposes of
tions. Indeed, the Kenya Situation Analy- cal procedures were used in the provision this analysis. In all three countries, 98–99%
sis yielded evidence that providers’ per- of particular contraceptives). of clients who were observed and inter-
formance increased during the first three As the client left the facility after her viewed had data that could be linked from
days in a week of observation, then de- visit, an interviewer approached her to ask both instruments. A total of 1,858 family
clined, suggesting that it was not possible if she could talk with the woman about the planning clients are included in this analy-
to remain on “best behavior” indefinitely.8 visit and her satisfaction with the services sis—583 from Ecuador, 539 from Uganda
Observation of client-provider interac- she received. The interviewer explained and 736 from Zimbabwe.
tions is also limited in that it includes only that she did not work for the clinic; that Eleven of the 25 quality of care indica-
a part of the client’s visit and does not all responses would remain confidential; tors were measured by both observation
cover, for example, group counseling ses- and that the woman’s answers would in and exit interview (Figure 1). Of these, two
sions, where clients may receive impor- no way affect the services she received in (provider demonstrates good counseling
tant information. Moreover, this type of the future. Assuming that the client gave skills and client is empowered) were com-
data collection requires more skilled per- her consent, the interviewer proceeded to posite indicators, measured by aggregat-
sonnel than a standard interview, since the ask her a series of questions (which usu- ing responses to several questions; these
observer must have adequate clinical ally took about 20 minutes). are not included in the analysis because
background to judge whether procedures Three of the countries that participated comparable questions from both instru-
are performed correctly. And she must be in the QIQ field test (Ecuador, Uganda and ments for all components of the indicator
quick enough to record a series of events Zimbabwe) implemented both client- were not available for all three countries.
that often do not occur in the same order provider observations and client exit Five additional indicators of quality for
as they are listed on the data collection interviews. The instruments used in the which questions appeared on both in-
form. three countries were almost identical, struments have been included, giving us
Exit interviews have their own set of although questionnaires for the exit a total of 14 indicators of quality of care
problems, the most serious of which is interviews were translated into local lan- for analysis.
courtesy bias. Respondents may give what guages. Clinically trained staff, nurses and Some questions (e.g., whether the
they consider socially acceptable answers, midwives conducted the client-provider provider gave instructions on when to re-
especially if they believe that the inter- observations in Uganda and Zimbabwe; turn) were virtually the same on the ob-
viewer works for the clinic or that unfa- physicians conducted observations in servation form and in the exit interview
Ecuador. In all three countries, social and were strictly objective. Others (e.g.,
*DISH, a project funded by USAID through a bilateral workers and sociologists conducted the whether the provider treated the client
agreement with the Uganda Ministry of Health, was de-
exit interviews. Data collection staff in with respect), while the same on both
signed to change reproductive-related behaviors by in-
creasing availability and improving the quality of inte-
each country, all of whom were female, forms, required a subjective judgment,
grated health services. The aim of SEATS, which is carried
underwent a one-week training session and answers could well differ between the
out by John Snow Inc. with funding from USAID, is to on the instruments and methodology; observer and the client. In yet other cases,
expand the development of, access to and use of quali- training also included a pilot test of the questions addressed a similar topic, but
ty family planning services in underserved populations. instruments. were not the same. For example, one item

64 International Family Planning Perspectives

on the observation form asked whether restarting a method after
Figure 1. Quality of care indicators measured by the Quick
the provider “gave accurate information not using it for more Investigation of Quality, by type of indicator, according to
on how to use the method.” The parallel than six months, switch- measurement instrument, 1998–1999
item in the exit interview asked the client ing methods or making
Indicator Client Obser- Facility
to provide correct information on how to their first visit to the fa- exit vations audit
use the method. However, if a client can- cility). However, the con- inter-
not provide the correct information, this traceptive methods they views

does not necessarily mean that the received differed sub- Provider
provider did not supply it during the visit. stantially. In Ecuador, the Demonstrates good counseling
skills (composite) 3 3
The first step of the analysis consisted IUD predominated, with
Assures client of confidentiality 3
of comparing frequencies on indicators 43% of clients receiving
that were available from both instruments a device during their Asks client about reproductive intentions 3 3
in all three countries. To simplify presen- visit; other frequently Discusses with client which method
tation of the data, we have organized the prescribed methods she would prefer 3 3
indicators into five of the six elements of were the injectable (21%) Mentions STDs or HIV/AIDS
and the pill (17%). In (initiates or responds) 3 3
the Bruce-Jain framework. (No indicators
available from both instruments captured Uganda, 71% of clients Discusses dual method use 3 3
technical competence.) received the injectable Treats client with respect/courtesy 3 3
We calculated simple agreement on each and 22% the pill. In Zim- Tailors key information to the client’s
indicator as the proportion of responses babwe, most clients re- particular needs 3
in which the observation and exit inter- ceived the pill (62%) or Gives accurate information on
view results were in agreement. Kappa co- the injectable (35%). the method accepted 3 3
efficients were calculated to correct for the Gives instructions on when to return 3 3
proportion of responses that would be in Interpersonal Relations Follows infection control procedures
agreement as a result of chance alone. Virtually all clients in outlined in guidelines 3
Since kappa values become low when the every country were Recognizes/identifies contraindications
prevalence deviates from 50%, and many treated with respect consistent with guidelines 3
of the indicators were highly skewed to- (Table 1, page 66), and Performs clinical procedures according
ward positive responses, we report kappa results on this indicator to guidelines 3
coefficients adjusted for prevalence and were highly consistent Other staff
bias.10 Kappa coefficients ranging from between observations Treat clients with dignity and respect 3
0.00 to 0.39 indicate poor agreement, the and client exit inter-
0.40–0.74 range indicates fair to good views (kappas, 0.98– Client Participates actively in discussion and selection
agreement, and values of 0.75–1.00 indi- 0.99). Results regarding of method, is “empowered” (composite) 3 3
cate excellent agreement.11 whether counseling and Receives her method of choice 3 3
Finally, we combined data from all three the pelvic examination
Believes the provider will keep her
countries and present percentage agree- took place in private information confidential 3
ment and kappa coefficients. We also as- were also similar on
sessed evidence of bias or systematic error both instruments; con- Facility
Has all (approved) methods available 3
using McNemar’s test for bias. Bias was sistency across instru-
Has basic items needed for delivery of
considered to be present if one instrument ments was good to ex- methods available through service delivery point 3
consistently rated the indicator higher (or cellent for Ecuador and
Offers privacy for pelvic
lower) than did the other instrument. Zimbabwe (kappas, examination/IUD insertion 3 3 3
0.74–0.94), and was Has mechanisms to make programmatic
Results lower but still good in changes on the basis of client feedback 3
Client Characteristics Uganda (0.63–0.65). Has received a supervisory visit
Clients’ characteristics, which may influence Where disagreement oc- in past _ months 3
their ability to accurately report informa- curred, clients were typ- Has adequate storage for contraceptives
tion from the visit, varied somewhat among ically less likely than ob- and medicines 3
the three countries. Overall, almost one-half servers to report that Has state-of-the-art clinical guidelines 3
of the women were aged 24–35, and age pat- privacy was adequate. Has acceptable waiting time 3 3
terns were similar across countries. Educa- In Ecuador, for example,
tional levels, however, varied; they were 99% of observers record- Notes: The two composite indicators were not included in the analysis because measures that
they are based on differed across countries. Five indicators that do not appear on this list were
highest in Ecuador, where 67% of clients had ed that the pelvic exam- also included in the study: provider sees client in privacy for counseling, asks client if she has
attended at least secondary school, and low- ination was conducted any concerns or problems, tells client how to use the method, gives information on side ef-
fects and explains that the method does not protect against HIV/AIDS.
est in Uganda, where only 40% of clients in privacy, compared
had advanced beyond primary school. with 93% of clients.
Reasons for coming to the clinic were Providers were supposed to ask re- was fair to good (kappas, 0.54 and 0.61, re-
similar. Slightly more than one-quarter of turning clients whether they had any con- spectively). In Ecuador, observers noted
clients in each country were new family cerns or problems. In Ecuador and Ugan- that the providers asked 84% of clients if
planning clients (defined as those who da (Zimbabwe had no data on this they had any problems or concerns,
were either coming to the facility for a fam- indicator), consistency between respons- whereas 87% of respondents answered af-
ily planning method for the first time, es from observations and exit interviews firmatively in exit interviews. In Uganda,

Volume 27, Number 2, June 2001 65

Monitoring Quality of Care in Family Planning Programs: Observations and Client Exit Interviews

Table 1. Percentage of visits exhibiting quality care, by measurement instrument; percentage agreement between instruments; and kappa
coefficients indicating strength of agreement—all by country, according to quality of care indicator

Indicator Ecuador (N=583) Uganda (N=539) Zimbabwe (N=736)

Obser- Interviews Agree- Kappa Obser- Interviews Agree- Kappa Obser- Interviews Agree- Kappa
vations ment vations ment vations ment

Interpersonal relations
Provider treats client with respect/courtesy 99 99 99 0.99 99 99 99 0.98 99 99 99 0.98
Provider sees client in private for counseling 99 87 88 0.74 87 88 82 0.63 97 95 92 0.85
Facility offers privacy for pelvic examination/
IUD insertion* 99 93 93 0.85 95 82 82 0.65 98 98 97 0.94
Provider asks client if she has any concerns
or problems† 84 87 76 0.54 87 86 81 0.61 u u u u

Choice of methods
Provider discusses client’s fertility intentions‡ 53 63 62 0.23 63 53 63 0.26 40 35 70 0.40
Provider discusses with client which method
she would prefer§ 99 98 98 0.95 74 85 69 0.38 90 94 87 0.76
Client receives her method of choice§ 80 84 91 0.82 76 81 82 0.64 89 87 95 0.88

Information given clients

Provider tells the client how to use the method** 85 97 85 0.70 94 93 89 0.77 85 82 83 0.64
Provider gives accurate information on
the method accepted** 84 75 75 0.50 94 100 94 0.78 85 96 83 0.66
Provider gives information on side effects** 71 80 75 0.49 86 77 78 0.57 70 65 70 0.41

Appropriate constellation of services

Provider mentions STDs or HIV/AIDS 13 27 79 0.57 22 30 69 0.38 12 15 84 0.68
Provider explains that method does not
protect against HIV/AIDS†† 19 34 73 0.46 40 59 63 0.27 10 52 55 0.08
Provider encourages dual method use†† 19 36 74 0.48 26 49 62 0.25 46 57 79 0.62

Mechanisms for continuity

Provider gives instructions on when to return 94 96 91 0.81 94 94 91 0.81 83 72 70 0.41

*Clients receiving a pelvic examination or IUD insertion: Ecuador (N=403), Uganda (N=34) and Zimbabwe (N=138). †Returning clients only: Ecuador (N=353) and Uganda (N=300). ‡New clients only:
Ecuador (N=177), Uganda (N=137) and Zimbabwe (N=198). §New clients with a method preference: Ecuador (N=109), Uganda (N=88) and Zimbabwe (N=150). **New clients who received a method:
Ecuador (N=145), Uganda (N=111) and Zimbabwe (N=191). ††New clients who received a nonbarrier method: Ecuador (N=130), Uganda (N=109) and Zimbabwe (N=169). Notes: u=unavailable. Kappa
coefficients, which correct for the proportion of responses that would agree because of chance alone, are adjusted for prevalence and bias.

the proportions were 87% and 86%, da, however, consistency was poor: Sixty- providers gave clients correct information
respectively. nine percent of responses were in agree- on how to use their selected method.
ment on this indicator, and the kappa For example, to receive a check for this
Choice of Methods value was 0.38. Rephrasing of the question item, providers must have told clients
To assist new clients in selecting the most in the exit interview in Uganda may have receiving the pill that it has to be taken
appropriate family planning method, led to the inconsistent responses. every day.
providers should ask them about their fer- In Ecuador and Uganda, the proportion For the indicator on whether the
tility intentions. Observers noted whether of women who stated during exit inter- provider told new clients how to use the
the provider and client discussed her de- views that they received their preferred method, consistency across instruments
sire for more children or the timing of the method (84% and 81%, respectively) was in all three countries was good to excel-
next birth; staff conducting exit interviews slightly higher than the proportion record- lent (kappas 0.64–0.77). In Ecuador, almost
asked each woman if the provider asked ed during observations (80% and 76%, re- all discrepant responses were for clients
her whether she would like to have more spectively). Results were more similar in for whom observers recorded that the in-
children. In each country, results from Zimbabwe: 89% from observations and formation was not provided but who re-
observations and exit interviews were 87% from exit interviews. Agreement was ported during exit interviews that they re-
comparable (53% and 63% in Ecuador, for excellent in Ecuador and Zimbabwe (kap- ceived this information. Information on
example), but agreement between instru- pas, 0.82 and 0.88, respectively), and good how to use the method may have been
ments was poor (kappas, 0.23–0.40). The in Uganda (0.64). given to clients during a supplemental
lack of agreement may have stemmed counseling session not covered by the
from differences in the items used to cal- Information Given Clients client-provider observation. No such pat-
culate the indicator: While the observation Whether clients received information on tern for discrepant responses was appar-
guide contained two items that captured how to use their chosen method was gath- ent in Uganda and Zimbabwe.
the ideas of both spacing and limiting, the ered in two ways during the exit inter- In Uganda and Zimbabwe, the propor-
exit interview asked only about limiting. view. Clients were first asked whether the tions of new clients whom observers con-
Consistency on whether the provider provider told them how to use the sidered to have received accurate infor-
discussed the client’s preferred method method. They were also asked questions mation on using their method (94% and
during the visit was excellent for Ecuador about how the method is used, to assess 85%, respectively) were lower than the
and Zimbabwe (kappas, 0.95 and 0.76, re- whether they had correct information proportions who could accurately re-
spectively), and very similar frequencies about it. For example, pill users were spond to the interview question on how
on responses were found from the two in- asked, “How often do you take the pill?” to use the method (100% and 96%, re-
struments within each country. In Ugan- By contrast, observers noted only whether spectively). The opposite was true in

66 International Family Planning Perspectives

Ecuador, where observers recorded that
Table 2. Percentage of visits exhibiting quality care, by measurement instrument; percentage
84% of clients were told how to use the agreement between instruments; kappa coefficients indicating strength of agreement; and
method, but only 75% of clients could cor- assessment of bias—for all three countries combined, by quality of care indicator
rectly answer the question posed during
Indicator Obser- Inter- Agree- Kappa Evidence
the exit interview. Agreement ranged from vations views ment of bias*
fair to excellent, depending on the coun-
Provider actions with client (objective)
try. A number of reasons could explain the Provider discusses with client which method
lack of consistency. For example, differ- she would prefer† 88 93 85 0.71 No
ences may reflect knowledge acquired Provider gives instructions on when to return 89 86 83 0.66 Yes
Provider asks client if she has any
during a previous visit (particularly concerns or problems‡ 86 86 79 0.57 No
among returning clients in Uganda or Provider discusses client’s fertility intentions§ 51 49 65 0.30 No
Zimbabwe obtaining the pill or injectable). Information given to client (objective)
Alternatively, they may be associated with Provider tells client how to use the method** 87 90 85 0.69 No
the amount or type of knowledge required Provider mentions STDs or HIV/AIDS 15 23 78 0.56 Yes
Provider gives information on side effects** 74 73 74 0.48 No
for a particular method. Perhaps clients Provider encourages dual method use†† 32 48 74 0.47 Yes
in Ecuador either did not know to check Provider explains that method does not
protect against HIV/AIDS†† 21 48 63 0.26 Yes
IUD strings or were too embarrassed to Provider gives accurate information on
mention this during the exit interview. It the method accepted** 87 91 84 0.67 Yes
should also be noted that these questions
Interpersonal relations (subjective)
are fundamentally different from the truly Provider treats client with respect/courtesy 99 99 99 0.98 No
paired questions; therefore, a lack of con- Client receives her method of choice† 83 84 90 0.80 No
Provider sees client in private for counseling 95 91 88 0.75 Yes
sistency may not be surprising. Facility offers privacy for pelvic
A comparison of the results on whether examination/IUD insertion‡‡ 99 94 93 0.86 Yes
new clients received information on the *Based on McNemar’s test for bias, p<.05. †New clients with a method preference (N=347). ‡Returning clients (N=653). §New clients
side effects of their selected method shows only (N=512). **New clients who received a method in Ecuador (N=447). ††New clients who received a nonbarrier method (N=408).
‡‡Clients receiving a pelvic examination or having an IUD inserted (N=575). Note: Kappa coefficients, which correct for the proportion
only fair agreement between observations of responses that would agree because of chance alone, are adjusted for prevalence and bias.
and exit interviews in each country (kap-
pas, 0.41–0.57). The level of agreement
ranged from 70% to 78%; the spread be- Ecuador, information was provided in a we reflect instances in which the observers
tween instruments was approximately 10 separate counseling session (either one-on- did not mark that this information was
points in Ecuador and Uganda, and five one or in groups) conducted by social work- given yet the clients reported receiving it.
points in Zimbabwe. ers and health educators prior to clients’ vis- In Uganda and Ecuador, similar pat-
its with the provider. In Uganda, about 50% terns were found for whether the provider
Appropriate Constellation of Services of new clients attended group talks that encouraged dual method use, while in
The QIQ instruments also captured top- covered family planning methods and the Zimbabwe, differences were much small-
ics other than family planning that were prevention of HIV and other STDs before er than on the previous indicator. Again,
discussed during counseling sessions. Fre- seeing the provider. Group talks were also most of the discrepant results (more than
quencies from exit interviews on these in- a frequent occurrence at facilities in Zim- 75%) reflected negative responses on the
dicators were generally higher than those babwe. As the client-provider observation observation and positive responses in the
from observations. In Ecuador, observers did not include information given to clients exit interview. This indicator, too, may
reported that 13% of clients received in- in these other settings, it is not surprising have been affected by information on STD
formation on HIV or other sexually trans- that the frequencies for the indicators mea- prevention that clients received in coun-
mitted diseases (STDs), whereas 27% of suring whether information was provided seling sessions and group talks that were
clients said they received such informa- during the visit are higher on the exit in- not covered by the observations.
tion; in Uganda, these proportions were terview than what was found during the
22% and 30%, respectively. In Zimbabwe, observation. Mechanisms for Continuity
results from both instruments were simi- For new clients, we also compared indi- An important indicator for continuity of
lar: 12% from observations and 15% from cators that measured whether the provider care is whether providers give clients any
exit interviews. Agreement ranged from mentioned that the accepted method (other instruction regarding their return to the
poor to fair (kappas, 0.38–0.68). The ma- than condoms) does not protect against facility. Agreement on this indicator from
jority of discrepant responses are for HIV, and whether she encouraged dual the two instruments was excellent for
clients who were recorded as not receiv- method use. In all three countries, the fre- Ecuador and Uganda (kappa, 0.81 for
ing information on HIV and other STDs quencies of positive responses for both in- each), where providers discussed return
during observation, but who reported re- dicators were higher on the exit interview visits with nearly all clients. In Zimbab-
ceiving this information when asked dur- than on the observation. The spread for the we, agreement on this indicator was fair
ing the exit interview. Note that this indi- first indicator was about 15 points in (0.41), and observers somewhat more fre-
cator captures only whether the topic was Ecuador and Uganda, and more than 40 quently than clients said that such a dis-
discussed and does not explore the con- points in Zimbabwe. Agreement between cussion took place (83% vs. 72%).
tent of the discussion. the two instruments was fair in Ecuador
Evidence from the fieldwork suggested (kappa=0.46), poor in Uganda (0.27) and Indicator Agreement by Question Type
that clients received information during very poor in Zimbabwe (0.08). Approxi- In Table 2, we present data for all three
their visit to the health facility from other mately 75% of the discrepant results in countries combined. For this table, we
sources in addition to the provider. In Ecuador and Uganda and 97% in Zimbab- have reorganized the indicators to reflect

Volume 27, Number 2, June 2001 67

Monitoring Quality of Care in Family Planning Programs: Observations and Client Exit Interviews

Figure 2. Kappa coefficients indicating the comparability of 14

ments—whether the major reason for these discrepancies was
indicators measured through observation and client exit interviews provider gave the client that clients received information from
accurate information on sources other than the observed client-
how to use the method provider interaction. Such other sources
Provider explains that method
does not protect against HIV/AIDS she chose. On this indi- as group talks and supplemental coun-
Provider discusses client’s fertility cator, we saw some evi- seling sessions need to be taken into con-
intentions dence of better results sideration in interpreting the results of this
Provider encourages dual method from the exit interview, study and in using these instruments in
use primarily because often the future.
Provider gives information on side in Uganda and Zimbab- The consistently high ratings for indi-
effects
we, new clients correctly cators measuring interpersonal relations
Provider mentions STDs or
HIV/AIDS reported key information may reflect that providers were on their
Provider asks client if she has any
on how to use their cho- best behavior, since they knew they were
concerns or problems sen method, yet ob- under study (the Hawthorne effect). This
Provider gives instructions on servers did not record upward bias, however, should have af-
when to return that accurate information fected the responses from observations
Provider gives accurate informa- was provided. While a and exit interviews equally—i.e., ob-
tion on the method accepted
client’s knowledge of her servers would have recorded better be-
Provider tells client how to use the
method
method may have been havior on the part of providers, and clients
Provider discusses with client
obtained during the visit would have reported the same during exit
which method she would prefer with the provider, in interviews. (Whether the high ratings on
Provider sees client in private for many cases, she may these subjective measures are due to the
counseling have already had correct Hawthorne effect cannot be addressed
Client receives her method of information or obtained with the current data.) Moreover, despite
choice it from other sources at the presence of observers, many objective
Facility offers privacy for pelvic the health facility. indicators suggest serious deficits in qual-
examination/IUD insertion
The third set of indi- ity in many areas.
Provider treats client with respect/
courtesy cators measure inter- A major drawback of exit interviews is
personal involvement; courtesy bias. One would expect that in-
0.0 0.2 0.4 0.6 0.8 1.0 we deemed these to in- dicators measuring subjective states such
volve more subjectivity. as attitude, opinions or feelings would be
Surprisingly, agreement more susceptible to courtesy bias than
the type of question and degree of com- between results from observations and more objective measures. We did not find
parability of the questions between the in- exit interviews was excellent for all of this to be true in our data. In fact, agree-
struments. In addition to measures of these (kappas, 0.75–0.98), and was actu- ment was highest on the indicators that
agreement, we present an assessment of ally higher than for the more objective in- we considered more subjective. It is pos-
bias—i.e., whether one instrument con- dicators. The two indicators that assessed sible, however, that providers were on
sistently rated the indicator higher (or whether privacy was adequate revealed their best behavior because of the ob-
lower) than did the other instrument. bias: Fewer clients than observers report- servers’ presence, and clients were truth-
The first indicators are objective mea- ed adequate privacy for counseling or ex- fully reporting good interpersonal rela-
sures of the provider’s actions with the aminations. This difference may suggest tions. Whether the clients would have
client. Agreement was fair to good (kap- that observers’ perceptions of what con- been as truthful in a situation where
pas, 0.57–0.71) on three of these indicators stitutes privacy differ from clients’, pos- providers were rude or unresponsive is
and poor on the fourth (0.30). The only in- sibly because of observers’ familiarity not known.
dicator for which we found evidence of with the health care system and its norms. Other sources of error are also possible.
bias was whether the return visit was dis- Effective training of observers can reduce Recall bias may account for a client’s “for-
cussed. This finding reflects that in Zim- interrater reliability, but it cannot elimi- getting” that a specific instruction or par-
babwe, clients greatly underreported the nate this difference in perception. For the ticular information was provided during
occurrence of such discussions. remaining two indicators, the responses the visit. Given that the client was inter-
A second set of indicators measure the showed no evidence of bias. viewed immediately following the visit,
information exchange that occurred be- Kappa coefficients for the 14 indicators she may not have had time to think about
tween the client and provider on differ- for the three countries combined are pre- the session and process all of the infor-
ent topics; we also considered these to be sented in Figure 2. Agreement ranged from mation that she received. This can be seen
objectively measured. All but one of these poor to excellent. Both kappa coefficients with indicators that are measured simi-
indicators had fair to good agreement and percentage agreement (which ranged larly in both instruments: Whether the
(kappas, 0.47–0.69). As we noted earlier, from 63% to 99%—see Table 2) gave very provider discussed the return visit with
clients frequently reported receiving in- similar findings for the indicators. the client is an example. Though the ques-
formation not recorded during the obser- tions are relatively straightforward, clients
vation, probably because this occurred Discussion sometimes reported that this was dis-
during an unobserved part of the visit. Overall, the results obtained from obser- cussed while it was not, and vice versa.
This set of indicators includes one that vations and client exit interviews were We also considered whether interviewee
was calculated from questions that were highly comparable for most indicators. To fatigue and a desire to terminate the in-
less than comparable on the two instru- the extent that discrepancies occurred, the terview quickly may have introduced two

68 International Family Planning Perspectives

types of error. A client may have provid- observations and exit interviews were ex- Indicators, New York, 1990.
ed any response to hurry along the inter- tremely close. As a monitoring tool, either 4. Bertrand J, Magnani RJ and Knowles J, Handbook of In-
view, resulting in an increase in discordant method could be used to calculate many of dicators for Family Planning Program Evaluation, Chapel
responses in the latter half of the exit in- these indicators—as long as a distinction Hill, NC, USA: Carolina Population Center, 1994.

terview. Or she may have provided what is made about the source of information, 5. Miller R et al., The Situation Analysis Approach to As-
she thought was the correct response (often the main cause of discrepancies in this sessing Family Planning and Reproductive Health Services:
A Handbook, New York: Population Council, 1997.
a yes), in hopes of quickly terminating the study. Our results show that similar con-
interview; this would have resulted in a clusions on the quality of care may be 6. Bertrand JT, Sullivan T and Rice J, Quick investiga-
tion of quality: measuring quality of care in family plan-
bias toward more positive responses in the reached regardless of the data collection in- ning programs, Chapel Hill, NC, USA: Carolina Popu-
second half of the interview. After exam- strument and methodology used. Where lation Center, 2001; and Sullivan T and Bertrand J,
ining the data from the three countries, we inconsistencies occur, judgment on the Monitoring quality of care in family planning programs
found no relationship with percentage “correct” answer will often have to be made by the quick investigation of quality (QIQ): country re-
agreement on the instruments and after taking local conditions into account. ports, MEASURE Evaluation Technical Report, Chapel
Hill, NC, USA: Carolina Population Center, 2000, No. 5.
whether the questions appeared earlier or Given the comparability of many of the
later in the interview. Neither did we see indicators, it could be argued that there is 7. Askew I, Mensch B and Adewuyi A, Indicators for
measuring the quality of family planning services in
evidence of more biased responses if the no need to use both observations and exit Nigeria, Studies in Family Planning, 1994, 25(5):268–283;
questions appeared in the latter half of the interviews; programs may reduce the costs and Huntington D, Miller K and Mensch B, The reliability
exit interview. and complexity of data collection by im- of the situation analysis observation guide, Studies in Fam-
Variations in clients’ characteristics are plementing only one of the two instru- ily Planning, 1996, 27(5):277–282.
another potential source of error that may ments. For example, a program that fo- 8. Ndhlovu L, Quality in the context of Kenya’s inte-
explain differences in agreement by coun- cuses its efforts on improving providers’ grated reproductive health services, dissertation, Uni-
try. Uganda, which had the lowest per- interpersonal skills as a way to increase versity of London, London, 1999.

centage agreement for all of the indicators, client satisfaction may opt to implement 9. Williams T, Schutt-Ainé J and Cuca Y, Measuring fam-
also had the clients with the lowest levels only the exit interview. Alternatively, a pro- ily planning service quality through client satisfaction
exit interviews, International Family Planning Perspectives,
of education. In an analysis not present- gram that wants to assess the information 2000, 26(2):63–71.
ed in this article, we found that agreement given to clients by the provider during
10. Byrt T, Bishop J and Carlin JB, Bias, prevalence and
on many indicators was slightly lower counseling and clinical examination may kappa, Journal of Clinical Epidemiology, 1993, 46(5):423–429.
among clients who had not attended sec- choose to implement only the observation.
11. Fleiss JL, Statistical Methods for Rates and Proportions,
ondary school than among those who had. The QIQ, however, was designed to
second ed., New York: John Wiley and Sons, 1981.
However, this is not sufficient to explain capture a short list of quality indicators for
the discrepancies seen. monitoring family planning programs,
A final consideration was whether the and the combined use of its three data col- Resumen
stigma associated with STDs and HIV lection instruments is recommended so Contexto: Es importante controlar la calidad
may have prohibited some clients from that the full set of indicators can be mea- de atención de los programas de planificación
mentioning that these topics were dis- sured. Selecting only the client-provider familiar, pero la complejidad y subjetividad del
cussed during the session. Results from observation would eliminate indicators tema presenta muchos desafíos.
this analysis do not support this potential that capture clients’ perspective on the Métodos: El sistema de Quick Investigation
bias. A larger proportion of clients than of care they receive. Selecting only the exit of Quality (QIQ) fue diseñado para controlar
observation records reported that infor- interview, on the other hand, would not la calidad de la atención de los servicios de pla-
mation on these topics was discussed, be- permit an assessment of the provider’s nificación familiar mediante la observación de
cause clients may have remembered re- technical competence during counseling las relaciones entre el proveedor y la cliente,
ceiving information in group talks and and clinical examination. Neither of these entrevistas a las clientes al final de su visita,
previous counseling sessions. permits assessment of the indicators that y la auditoría de las instalaciones. El QIQ fue
As we noted previously, the measure- the facility audit (not discussed in this ar- probado in situ en muchos países, en 1998 y
ment of quality can be difficult because of ticle) measures: factors that influence the 1999. Mediante el uso de datos comparables
the complexity and subjectivity of the facility’s readiness to provide quality ser- de 583 clientes del Ecuador, 539 de Uganda y
topic. Given these difficulties, some error vices, such as supplies in stock, conditions 736 de Zimbabwe, este análisis examina las re-
in measurement of quality of care indica- of the facility and types of records kept. laciones de los resultados obtenidos mediante
tors is expected. This measurement error Therefore, although one instrument may observaciones y entrevistas.
is acceptable as long as it is minimized and be selected over another where resources Resultados: Con respecto a un indicador de-
inconsistencies can be reasonably ex- are limited, there is a cost in the breadth terminado, fueron muy diferentes los niveles
plained. Although agreement between in- of indicators that will be available to mea- de acuerdo entre los datos de las observacio-
struments was high for most indicators in sure quality. nes y entrevistas entre un país y otro, aunque
this study, it was poor for a few. While the al comparar estos datos dentro de cada país,
source of error could usually be explained, References los resultados fueron congruentes entre los di-
the inconsistencies underscore the need 1. Bruce J, Fundamentals of quality of care: a simple versos instrumentos. El nivel de acuerdo fue
framework, Studies in Family Planning, 1990, 21(2):61–90.
to understand the local context and im- entre bueno y excelente en los tres países com-
plementation of the instruments when in- 2. Simmons R and Elias C, The study of client-provider binados (kappas, 0.47–0.98) en 13 de los 14 in-
interactions: a review of methodological issues, Studies
terpreting the results. in Family Planning, 1994, 25(1):1–17.
dicadores examinados; las observaciones y las
The level of quality differed by country entrevistas realizadas presentaron resultados
3. Report of the Subcommittee on Quality Indicators in
on a number of indicators, but compara- Family Planning Service Delivery, submitted to U.S.
congruentes en el 63–99% de los casos. El
bility on the instruments for a given coun- Agency for International Development’s Task Force on acuerdo presentó su nivel más elevado entre
try was high. In many cases, results from Standardization of Family Planning Program Performance los indicadores que medían las relaciones per-

Volume 27, Number 2, June 2001 69

Monitoring Quality of Care in Family Planning Programs: Observations and Client Exit Interviews

sonales. Las incongruencias reflejaron prin- tion sont sources de nombreuses gageures. nés, la concordance s’est révélée bonne à ex-
cipalmente el hecho de que las clientes recibí- Méthodes: L’ERQ (enquête rapide de quali- cellente (kappas, 0,47–0,98) sur 13 des 14 in-
an información por fuera de la interacción ob- té) a été mise au point pour contrôler la qua- dicateurs examinés; les observations et les en-
servada entre clientes y proveedores. lité des prestations sur la base de l’observation trevues à la sortie ont produit des réponses
Conclusiones: Las observaciones y las en- des rapports entre clientes et prestataires, d’en- convergentes dans 63% à 99% des cas. La
trevistas a las clientes ofrecieron resultados si- trevues organisées à la sortie des clientes et concordance s’est avérée la plus élevée sur les
milares con respecto a muchos indicadores. Sin d’audits d’établissement. L’ERQ a été testée indicateurs de mesure des relations interper-
embargo, en tanto que algunos programas sur le terrain, dans plusieurs pays, en sonnelles. Les divergences reflétaient princi-
optan por un instrumento en particular en 1998–1999. Au départ des données liées de 583 palement la réception, par les clientes, d’in-
lugar de otros debido a las restricciones de re- clientes en Équateur, 539 en Ouganda et 736 formations obtenues en dehors du rapport
cursos, la combinación ofrece una evaluación au Zimbabwe, cette analyse examine la com- cliente/prestataire observé.
más compleja con respecto a la calidad de la parabilité des résultats obtenus des observa- Conclusions: Les observations et entrevues
atención. tions et des entrevues à la sortie. de clientes à la sortie produisent des résultats
Résultats: Pour un indicateur donné, les ni- fort comparables pour de nombreux indica-
Résumé veaux de concordance entre les données des teurs. Si leurs ressources limitées imposent à
Contexte: Le contrôle de la qualité des pres- observations et des entrevues variaient d’un certains programmes le choix d’un instrument
tations offertes dans le cadre des programmes pays à l’autre. Au sein d’un même pays, les ré- seulement, on soulignera toutefois que la com-
de planning familial est, certes, important, sultats étaient toutefois cohérents d’un in- binaison des deux permet une meilleure éva-
mais la complexité et la subjectivité de la ques- strument à l’autre. Pour les trois pays combi- luation de la qualité des prestations.

70 International Family Planning Perspectives