Symposium: Challenges in Targeting Nutrition Programs

Methodological Challenges in Performing Targeting: Assessing Dietary Risk

for WIC Participation and Education
1
Laura E. Cauleld
2
Center for Human Nutrition, The Johns Hopkins University, Bloomberg School of Public Health,
Baltimore, MD 21205
ABSTRACT This paper summarizes recent evaluations of indicators of dietary risk used to determine eligibility
and nutrition education and counseling within the Special Supplemental Nutrition Program for Women, Infants, and
Children (WIC). The concept of dietary risk relates to an inadequate dietary pattern or a failure to adhere to the food
guide pyramid and, within WIC, is typically assessed based on an individuals intake from a 24-h recall or FFQ.
Available evidence suggests, however, that even with a high-quality technique, substantial error exists in these data
at the level of the individual, making the likelihood of misclassication high. Such data, with appropriate statistical
procedures, can provide valid information at the group or the population level, and this is a future area for indicator
development and incorporation into the management of WIC and other nutrition education programs addressing
dietary risk. J. Nutr. 135: 879881, 2005.
KEY WORDS: targeting nutrition program diet nutrition risk indicator
There is a long history of programs designed to improve the
nutrition and the health of vulnerable populations in the
United States and throughout the world. Increasingly, there is
interest in evaluating the effectiveness of these programs to
document impact and to improve performance, as well as
pressure to provide evidence for continued or expanded fund-
ing. Early evaluations of nutrition programs (1,2) did not
demonstrate the impact desired by many in the scientic and
programmatic communities, and, since then, a literature has
emerged to address many of the methodological issues that
may explain less than desired impact. Principal among these
issues has been the idea that nutrition programs could be more
effective if they were better targeted so that they would reach
only those individuals who truly need or would benet from
the services offered (3). This can be accomplished if informa-
tion is available that adequately distinguishes among individ-
uals for this purpose; that is, information that screens individ-
uals and correctly classies who should or should not receive
program services. Screening for entry into nutrition programs
based on income or nutritional risk is common, and, more
recently, there have been reviews of the progress made to date
on the use of screening indicators for one of the largest
nutrition programs in the United States, the Women, Infants,
and Children (WIC)
3
Program. The purpose of this paper is to
use this information to highlight key methodological issues for
effectively targeting programs with nutritional outcomes.
The WIC program and the 1996 and 2002 Institute of
Medicine reports
The Special Supplemental Nutrition Program for WIC
began in 1972. The intent of the WIC program is to support
and strengthen health care during critical periods of growth
and development, to prevent health problems, and to improve
health status. To that end, it provides supplemental foods,
nutrition education, and health referral services to low-income
pregnant or postpartum women, infants, and young children.
Currently, it is estimated to serve about 8 million participants
per month (4).
The WIC program is a grant program for which funding
limits are set annually by the U.S. Congress. In addition to
categorical and income eligibility criteria, individuals must
show evidence of some form of nutrition risk based on either
anthropometric, biochemical, medical, or dietary factors.
These criteria allow for the prioritization of individuals based
on health risk and the potential to benet from the program in
the event of limitations in funding. For example, low-income
pregnant women and infants with anthropometric or hemato-
logical risk (e.g., low weight, low BMI, anemia) are given the
highest priority, whereas low-income pregnant or postpartum
1
Presented as part of the symposium Challenges in Targeting Nutrition
Programs given at the 2004 Experimental Biology meeting on April 20, 2004,
Washington, DC. The symposium was sponsored by the American Society for
Nutritional Sciences. The proceedings are published as a supplement to The
Journal of Nutrition. This supplement is the responsibility of the Guest Editors to
whom the Editor of The Journal of Nutrition has delegated supervision of both
technical conformity to the published regulations of The Journal of Nutrition and
general oversight of the scientic merit of each article. The opinions expressed in
this publication are those of the authors and are not attributable to the sponsors
or the publisher, editor, or editorial board of The Journal of Nutrition. The Guest
Editors for the symposium publication are Edward A. Frongillo and Jean-Pierre
Habicht, Division of Nutritional Sciences, Cornell University.
2
To whomcorrespondence should be addressed. E-mail: lcaule@jhsph.edu.
3
Abbreviatons used: IOM, Institute of Medicine; Se, sensitivity (Se); Sp,
specicity (Sp); WIC, Womens Infants and Children.
0022-3166/05 $8.00 2005 American Society for Nutritional Sciences.
879

b
y

g
u
e
s
t

o
n

A
p
r
i
l

3
,

2
0
1
1
j
n
.
n
u
t
r
i
t
i
o
n
.
o
r
g
D
o
w
n
l
o
a
d
e
d

f
r
o
m

women and low-income children who are at health risk due to
a poor diet receive lower priority for WIC participation.
The performance of such a system rests on the quality of the
information used to evaluate and classify individual women
and children within the prioritization scheme. In the last 10 y,
2 separate Institute of Medicine (IOM) committees have eval-
uated the performance of the indicators used for this screening
process (5,6). Here our focus is on issues raised in the second
report regarding the ability to screen individuals with respect
to the criteria, dietary risk, which refers to dietary decien-
cies that impair or endanger health, such as inadequate dietary
patterns assessed by a 24-h dietary recall, dietary history, or
food frequency checklist [Code of Federal Regulations Sub-
part C, Section 246.7(e) (iii) (2)]. Operationally, this has been
extended to 2 certication criteria, inadequate diet and
failure to meet the dietary guidelines.
The nature of dietary risk
The construct, dietary risk is complex and not open to
direct measurement. An individuals risk results from multiple
individual decisions made daily over extended periods of time.
For example, whether an individuals usual fat intake is low,
average, or high depends to a great extent upon the frequency
with which they have days of high-fat intake and that on many
days, the fat intakes of these individuals will be similar. The
nature of the decision process, the choices available, and
multiple other inuences lead to the fact that dietary intakes
are inherently unstable from day to day within individuals,
and, even in circumstances of reduced resources, dietary in-
takes are subject to substantial within-subject variability. De-
pending on the nutrient, within-subject variability may be 16
times greater than among-subject variability (7,8). Assessing
the dietary intakes of children is further complicated, because
dietary patterns change as children age, and many individuals
(e.g., parents, child, daycare provider, school) usually deter-
mine a childs dietary intake.
Two distinct approaches have been taken to deal with this
variability and to characterize usual intake. First, one can
collect multiple days of intake and average the data as an
estimate of usual intake; the mean is considered to be an
accurate (unbiased) estimate of ones usual intake, with the
variance of the data indicating the precision of the derived
mean. Second, one can use an FFQ, in which the individual is
asked to address the variability by summarizing their usual
intake of specic food items, based on their knowledge of their
day-to-day dietary choices. It should be noted that one can use
either method to quantify usual intakes of foods or food groups
to assess dietary patterns or adherence to the food guide
pyramid or dietary guidelines; it seems, however, that within-
individual variability in food intake is similar or greater that
that for nutrient intakes.
Because of the inherent difculty in estimating usual di-
etary intakes, scientists have also focused attention on the
potential of what are called behavioral indicators. Behav-
ioral indicators do not measure diet per se but rather depict
factors that inuence food choices or dietary intakes (9). Such
factors could include consumption of specic foods or groups of
foods, specic meal patterns (percent of meals consumed
outside the home), health-related behaviors, psychosocial
characteristics, family food practices, and ecological factors
related to the home, neighborhood, or community.
Indicator performance
Although one can identify a list of criteria with which to
evaluate the usefulness of indicators for screening or other
purposes (e.g., feasibility, cultural appropriateness), the hall-
mark criteria for indicator evaluation are accuracy and reli-
ability (10). Accuracy addresses whether an indicator is really
measuring what is intended vis-a`-vis some truth. Reliability
refers to the reproducibility of the measure over timethat is,
whether repeated measurements provide the same results.
Intraindividual variability in dietary intake is real, but its
presence acts as a form of random error and diminishes the
reliability of the measure of usual dietary intake. In the case of
recall data, multiple recalls would provide an accurate estimate
of usual intake, and information on the within-subject vari-
ance (s
w
2
) or within-subject coefcient of variation (CV
w
) can
be used in a common statistical formula to calculate the
number of days (replicates) needed to attain a desired level of
precision of that estimate (7,8). For example, for nutrients
with CV
w
of 3070%, 1540 replicates (days) would be
needed to estimate mean intake with a precision of 20%
around the true mean. Because the collection of this level of
replication may not be feasible, an important judgment must
be made with respect to the desired precision with which one
would like to make critical decisions, for example, regarding an
individuals WIC eligibility.
These results highlight the potential advantage of FFQs, if
they can provide valid and reliable information on an indi-
viduals usual intake. For FFQs, reliability is expressed in terms
of the reproducibility of the intake estimates when the FFQ is
administered on multiple occasions, and accuracy is evaluated
by comparisons of intakes with estimated intakes from multi-
ple records or dietary recalls. As reviewed by the IOM (6),
reliability coefcients for FFQ generally range from 0.7 to 0.9,
and accuracy coefcients generally range from 0.3 to 0.7 across
nutrients.
An emerging literature provides information on the reli-
ability and the accuracy of these behavioral indicators for
assessing dietary risk (6). Results vary widely across the indi-
cators. For example, the reliability can be near 0.0 for some
psychosocial indicators but be moderate (0.50.7) for specic
food pattern indicators. The accuracy of these indicators varies
as well, but few have accuracy coefcients 0.30.
For screening purposes, the WIC professional wishes to
estimate and to characterize the individual clients usual nu-
trient or food intake and decide whether or not they are at
dietary risk (eligible for WIC). Less than perfect measures of
dietary risk result in misclassication of individuals regarding
their true eligibility. The level of misclassication can be
captured by the sensitivity (Se) and the specicity (Sp) of the
indicator, with Se referring here to the ability of the indicator
to identify those truly at dietary risk, and Sp referring to the
ability of the indicator to identify those not at dietary risk. Less
than perfect Se means that truly eligible individuals may not
be classied as eligible and denied services, whereas less than
perfect Sp means that others who are truly not eligible for the
services may receive them. Walker and Blettner (11) calcu-
lated the probability of misclassication in dietary intake for a
given level of error in estimated dietary intake. As shown in
Table 1, for accuracy coefcients between 0.3 and 0.7, only 26
to 40% of individuals would classied into the correct quintile
of intake.
Implications and future directions
These results indicate that, even with rigorous measure-
ment techniques, there is substantial error in estimates of
individual dietary intakes using current methods. As discussed,
this error is attributed largely to the complex nature of the
dietary behaviors. Unfortunately, it leads to high levels of
SYMPOSIUM 880

b
y

g
u
e
s
t

o
n

A
p
r
i
l

3
,

2
0
1
1
j
n
.
n
u
t
r
i
t
i
o
n
.
o
r
g
D
o
w
n
l
o
a
d
e
d

f
r
o
m

misclassication of individuals with respect to WIC eligibility.
It must also be considered that some errors are more costly
than others. Whereas some may focus on the cost of ineligible
individuals obtaining services, others would argue that the cost
of missing individuals at risk is much higher, particularly in the
long run. It is the task of policy makers and the public to
decide how much and what type of misclassication error they
are willing to tolerate when certifying individuals to receive or
not receive federally funded WIC services; however, there is
likely consensus that misclassication rates of 50% would
not be tolerated.
The measures used to assess risk also serve as the foundation
for the provision of WIC services, including package tailoring
and individualized nutrition education or counseling. Given
the level of error in current individual assessment methods for
dietary intake, the foundation for effective nutrition education
and counseling is not present. New methods or indicators are
needed for screening and formulating education and counsel-
ing services.
This discussion has focused on making inferences about
individuals. Indicators can be used, however, to make infer-
ences about individuals or groups of individuals. This is an
important distinction, because the level of precision (lack of
random error) required to make inferences at the individual
level is much greater than for making inferences at the pop-
ulation level. When making inferences about groups (or at the
population level), it is still possible to derive unbiased esti-
mates of the underlying phenomena of interest, using well-
developed statistical procedures (12,13). For example, recalls
or FFQs can be used to characterize the usual dietary intakes or
food intake patterns of a target population, such as a WIC
clinic area. This is true because even if individual values in the
distribution are measured with error or are inherently unstable,
these errors tend to cancel each other out, and individual
values contribute information on the nature of the population
distribution. Thus, with methods currently in use, the WIC
program could characterize populations with respect to dietary
risk and use the information to provide the context for edu-
cation and counseling services. This was recommended by the
second IOM committee (6), and, since then, efforts have
begun to implement this shift in focus from individual to
population.
LITERATURE CITED
1. Beaton, G. H. & Ghassemi, H. (1982) Supplementary feeding pro-
grams for young children in developing countries. Am. J. Clin. Nutr. 35 (suppl.):
863916.
2. Abrams, B. A. (1993) Preventing low birth weight: does WIC work? A
review of evaluations of the special supplemental food program for women,
infants and children. Ann. N.Y. Acad. Sci. 678: 306316.
3. Habicht, J. P. & Pelletier, D. L. (1990) The importance of context in
choosing nutritional indicators. J. Nutr. 120: 15191524.
4. United States Department of Agriculture [Online]. http://www.fns.
usda.gov/pd/ [accessed April 2004].
5. Institute of Medicine (IOM) (1996) WIC Nutrition Risk Criteria: A Sci-
entic Assessment. Food and Nutrition Board (FNB), IOM. National Academy
Press, Washington, DC.
6. Institute of Medicine (IOM) (2002) Dietary Risk Assessment in the WIC
Program. Food and Nutrition Board (FNB), IOM. National Academy Press, Wash-
ington, DC.
7. Willet, W. (1998a) Nature of variation in diet. In: Nutritional Epidemi-
ology. Monographs in Epidemiology and Biostatistics, Vol. 15 (Willet, W., ed.), pp.
3349. Oxford University Press, Oxford, UK.
8. Nelson, M., Black, A. E., Morris, J. A. & Cole, T. J. (1989) Between- and
within-subject variation in nutrient intake from infancy to old age: estimating the
number of days required to rank dietary intakes with desired precision. Am. J.
Clin. Nutr. 50: 155167.
9. Baranowski, T. (1996) Psychosocial and sociocultural factors that
inuence nutritional behaviors and interventions: Cardiovascular disease. In: Be-
yond Nutritional Recommendations: Implementing Science for Healthier Popula-
tions (Garza, C., Haas, J., Habicht, J.-P. & Pelletier, D., eds.), pp. 163188. Cornell
University, Ithaca, NY.
10. Rothman, K. J. (1986) Modern Epidemiology. Little, Brown, Boston,
MA.
11. Walker, A. M. & Blettner, M. (1985) Comparing imperfect measures of
exposure. Am. J. Epidemiol. 121: 783790.
12. Traub, R. E. (1994) Reliability for the Social Sciences, Theory and
Applications. Sage Publications, Thousand Oaks, CA.
13. Willet, W. (1998b) Corrections for the effects of measurement error, In:
Nutritional Epidemiology Monographs in Epidemiology and Biostatistics, Vol. 15
(Willet, W., ed.), pp. 302320. Oxford University Press, Oxford, UK.
TABLE 1
Probabilities of correct classication of quintile of dietary
intake depending on the level of error in the indicator
1
Correlation coefcient (r)
0.8 0.7 0.6 0.5 0.4 0.3 0.2
Correctly
classied (%) 46.7 40.3 35.7 32.1 29.0 26.3 24.0
1
Adapted from Ref. 11.
METHODOLOGICAL CHALLENGES IN PERFORMING TARGETING 881

b
y

g
u
e
s
t

o
n

A
p
r
i
l

3
,

2
0
1
1
j
n
.
n
u
t
r
i
t
i
o
n
.
o
r
g
D
o
w
n
l
o
a
d
e
d

f
r
o
m

ERRATUM
WALTERC. RUSSELL,M. WRIGHTTAYLORANDJAMES V. DERBY,JR. The folie acid
requirement of turkey poults on a purified diet.
Journal of Nutrition vol. 34, no. 6, December, 1947. Page 632: second line from
top of page to be changed to
100 gm of purified diet (1.5 mg per kilo) for optimum growth

b
y

g
u
e
s
t

o
n

A
p
r
i
l

3
,

2
0
1
1
j
n
.
n
u
t
r
i
t
i
o
n
.
o
r
g
D
o
w
n
l
o
a
d
e
d

f
r
o
m

ERRATUM
Chakrabarty, Krishna, and Gilbert A. Leveille 1968 Influ
ence of periodicity of eating on the activity of various enzymes
in adipose tissue, liver and muscle of the rat. J. Nutr., 96: 76-
82. In tables 1, 2, and 3, pages 78 and 79, headings immediately
above columns of data should read milliunits/mg protein * in
stead of units/mg protein l.
b
y

g
u
e
s
t

o
n

A
p
r
i
l

3
,

2
0
1
1
j
n
.
n
u
t
r
i
t
i
o
n
.
o
r
g
D
o
w
n
l
o
a
d
e
d

f
r
o
m