Journal of Clinical Epidemiology 168 (2024) 111278

ORIGINAL RESEARCH

Grilling the data: application of specification curve analysis to red meat

and all-cause mortality
Yumin Wanga, Tyler Pitreb, Joshua D. Wallachc, Russell J. de Souzad, Tanvir Jassale,
Dennis Bierf, Chirag J. Patela, Dena Zeraatkarg,h,*
a
Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
b
Department of Medicine, McMaster University, Hamilton, Ontario, Canada
c
Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, USA
d
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
e
Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada
f
Department of Pediatrics, Baylor College of Medicine, Houston, TX, USA
g
Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada
h
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
Accepted 5 February 2024; Published online 12 February 2024

Abstract
Objectives: To present an application of specification curve analysisda novel analytic method that involves defining and implementing
all plausible and valid analytic approaches for addressing a research questiondto nutritional epidemiology.
Study Design and Setting: We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from
a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all
defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed.
We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the
effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic
specifications we sourced from primary studies.
Results: Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we
identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, contin-
uous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 partic-
ipants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to
or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a
median hazard ratio of 0.94 (interquartile range: 0.83e1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which
indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality.
Conclusion: We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an inno-
vative solution to analytic flexibility. Ó 2024 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-
NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Keywords: Nutrition; Red meat; All-cause mortality; Multiverse analysis; Specification curve analysis; Vibration of effects

consultant for Hagens Berman Sobol Shapiro LLP and Dugan Law Firm
Conflicts of interest: Dr Wallach reported receiving grant support from
APLC; and serving as a medRxiv affiliate.
the Food and Drug Administration, Arnold Ventures, Johnson & Johnson
Funding: None.
through Yale University, and the National Institute on Alcohol Abuse and
* Corresponding author. 1280 Main St. W, Hamilton, Ontario, Canada.
Alcoholism of the NIH under award 1K01AA028258; serving as a
E-mail address: zeraatd@mcmaster.ca (D. Zeraatkar).

https://doi.org/10.1016/j.jclinepi.2024.111278
0895-4356/Ó 2024 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/
licenses/by-nc-nd/4.0/).
2 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

Plain language summary

Randomized trials represent the optimal design for investigating the health effects of medical interventions. They
pose important challenges, however, when it comes to studying the health effects of food and nutrition. Hence, inves-
tigators commonly perform nutritional epidemiology studies. These studies are observational in design, collect infor-
mation from large groups of people, and look for patterns between their diet and their health.
There are, however, concerns about the trustworthiness of nutritional epidemiology studies, exemplified by cases
where they have produced inconsistent results. A growing body of evidence suggests that these inconsistent findings
may be explained by differences in analytic choices (ie, different ways of analyzing the same data). When investigators
analyze nutritional epidemiology studies (and other types of observational data), there are often hundreds of equally
justifiable ways of analyzing the data, each of which may produce different results. Hence, investigators may perform
several analyses and selectively report results for the analysis, that is, most interesting or publishable.
In this study, we apply a novel analytic methoddcalled specification curve analysisdto investigate the effect of red
meat on all-cause mortality. This method involves defining and implementing all plausible and justifiable analytic ap-
proaches for addressing a research question. Investigators can subsequently consider the range of all plausible results
and express more confidence in results that are consistent across all or most justifiable analytic specifications.
Our work suggests that specification curve analysis can be useful for studying the effects of diet on health. It provides
a practical and new way to deal with the challenge of analytic flexibility. Broader application of specification curve
analysis, along with other methods, may improve the credibility of such studies.

What is new?
1. Background
Key findings Unlike randomized trials for which investigators typi-
 The analysis of nutritional epidemiology data is cally register protocols and statistical analysis plans before
complex and there is often limited consensus the collection of any data, when investigators analyze data
among experts about the ideal approach. from observational studies, there are often hundreds of
equally justifiable ways of analyzing the data, each of
 While discrepancies in analytic models may result
which may produce results that vary in direction, magni-
from differences in opinions regarding the optimal
tude, and statistical significance [1e7]. The variability of
analytic approach among well-intentioned investi-
effect estimates due to alternative analytic approaches is
gators, some investigators may test many alterna-
called ‘vibration of effects’ [2]. Empirical evidence shows
tive analytic specifications and selectively report
that results from observational studies may be highly
results for the analysis that yields the most inter-
dependent on analytic choices [1e5].
esting findings.
While our empirical and theoretical understanding of the
question being investigated should guide our analytic
What this adds to what was known?
choices, our knowledge of complex biomedical and envi-
 We apply a novel analytic methoddcalled specifi-
ronmental systems is limited and even experienced investi-
cation curve analysisdto investigate the effect of
gators often come to different conclusions about the ideal
red meat on all-cause mortality. This method in-
analytic approach [4,6,8e13].
volves defining and implementing all plausible
While we anticipate that discrepancies in analytic
and justifiable analytic approaches for addressing
models often result from differences in opinions regarding
a research question.
the optimal analytic approach among well-intentioned in-
vestigators, some investigators may test many alternative
What is the implication and what should change
analytic specifications and, intentionally or unintentionally,
now?
selectively report results for the specification that yields the
 We show variability in results across plausible an-
most statistically significant or interesting results or results
alytic specifications.
that support their preconceived hypotheses. Evidence
This research demonstrates how specification curve shows that investigators’ prior beliefs and expectations in-
analysis can be effectively applied to nutritional fluence their results [5]. In the presence of strong opinions,
epidemiology, providing a practical and innovative investigators’ beliefs and expectations may shape the liter-
solution to the problem of analytic flexibility. ature to the detriment of empirical evidence [5].
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 3

include the type of analytic model (eg, Poisson regression,

Box 1 Specification curve analysis Cox proportional hazards model), choice of covariates (ie,
investigators studying the same question will consider
different adjusting variables [19]), operationalization of
When investigators analyze data from observational the exposure variable and covariates in the model (eg, trans-
studies, they may make numerous potentially
justifiable, but still subjective, analytic decisions on
formations, categorizations of continuous variables, func-
which the direction, magnitude, and statistical tional form), and methods to address missing data, among
significance of results may be contingent. Specification others [8]. Investigators often present several sensitivity an-
curve analysis may mitigate this issue [26]. alyses to investigate the effects of these uncertain analytic
Specification curve analysis involves defining and decisions on the results, but the choice of sensitivity ana-
implementing all plausible and justifiable analytic
methods for investigating a research question.
lyses is also subjective and investigators may be more in-
Investigators subsequently interpret the distribution of clined to report sensitivity analyses that affirm their
results across all plausible analyses, instead of focusing primary findings.
on the results of only one analysis. A large body of evidence shows inconsistency in the re-
The implementation of specification curve analysis sults of nutritional studies, some of which may be explained
involves:
by analytic flexibility [3,8,20,21]. Such inconsistencies
(1) Defining all plausible choices across all aspects of the have eroded trust in nutritional epidemiology and subjected
analysis. This typically includes: the field to criticism [22,23]. Nevertheless, nutritional
epidemiology studies continue to play a crucial role in
- Criteria for selecting eligible participants for inclusion in shaping dietary recommendations and policies, making it
the analysis
- Type of analytic model (eg, logistic, Poisson, or Cox pro-
imperative to draw credible inferences from these studies
portional hazards models) [14,15,24,25].
- Choice of covariates
- Operationalizations of the exposure variable and covariates
(eg, transformations, functional form) 1.2. Specification curve analysis

(2) Enumerating all justifiable combinations of these analytic

Specification curve analysisdsometimes called multi-
choices to produce a comprehensive list of all the ways in verse analysisdis a novel analytic method that involves
which the data may be reasonably analyzed. For example, defining and implementing all plausible and valid analytic
three unique choices for five aspects of the analysis yield approaches for addressing a research question [26] (Box 1).
243 unique analytic specifications (535). Through this approach, investigators define all plausible
(3) Implementing all or a random sample of all reasonable
analytic specifications.
and justifiable choices for all aspects of the analysis (eg,
(4) Ordering the effect estimates from all analyses based on choice of model, covariates, etc.), enumerate all justifiable
their direction and magnitude and presenting results on a combinations of these choices to produce a comprehensive
specification curve plot. A specification curve plot reports list of all the ways in which the data may be reasonably
the results of all analyses at the top and analytic charac- analyzed (i.e., analytic specifications), implement all or a
teristics at the bottom. The specification curve plot visually
communicates the distribution of results across all speci-
random sample of the valid analytic specifications, and
fications and the aspects of the analysis that are most draw inferences using the distribution of results from all
consequential in influencing the direction and magnitude plausible and justifiable specifications.
of findings. Specification curve analysis offers advantages to con-
ventional methods for data analysis. It allows investigators
to draw more credible inferences that are not contingent on
arbitrary analytic decisions and reduces the opportunity for
investigators to conduct many analyses and selectively
report results for analyses that yield the most interesting re-
sults, although it does not completely eliminate subjectivity
1.1. Nutritional epidemiology
in analytic decisions.
Nutrition is a field particularly amenable to analytic flex- While specification curve analysis has been previously
ibility [14]. Trials investigating the health effects of nutri- applied in psychology and economics, it has seldom been
tional exposures are often not feasible and so the applied in nutritional and environmental epidemiology
evidence is primarily comprised of nutritional epidemi- [5,27].
ology studiesdobservational studies that recruit large
groups of people and look for patterns between diet and
1.3. Objectives
health [15,16].
The analysis of nutritional epidemiology data is complex We apply specification curve analysis to investigate the
and there is often limited consensus among experts about effect of unprocessed red meat on all-cause mortalityda
the ideal approach [17,18]. Sources of analytic flexibility question that has yielded inconsistent results in the
4 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

literature and produced conflicting dietary recommenda- 2.2. Study population

tions. While this study may provide insights on the health
The National Health and Nutrition Examination Survey
effects of red meat, the primary objective is to demonstrate
(NHANES) is a repeated cross-sectional probability survey
the application of a novel analytic methoddspecification
by the US Centers for Disease Control and Prevention to
curve analysisdto nutritional epidemiology.
characterize the health and nutritional status of the nonin-
A critical limitation of specification curve analysis is the
stitutionalized, civilian US population [30]. The survey is
subjectivity involved in selecting justifiable analytic speci-
based on household interviews and physical examinations
fications. Investigators may disagree about justifiable ana-
and is representative of the US population by its survey
lytic approaches or may present results of analyses that
are only marginally justifiable. To mitigate this issue, our sampling method. The survey collects demographic, socio-
economic, dietary, and health-related data by household
analytic specifications were informed by the most common
interview, and medical, dental, physiological measure-
analytic methods used in previous published studies ad-
ments, and laboratory tests by physical examination.
dressing the effects of red meat on all-cause mortality.
For this analysis, we used the continuous 2007e2014
NHANES data linked with the National Death Index [31]
and the Food Patterns Equivalents Database. The National
2. Methods Death Index is a database established by the National Cen-
ter for Health Statistics that contains information on all
This study was exempt from institutional ethics review deaths in the United States. We extracted mortality status
because it uses secondary deidentified data. We report our from the National Death Index up to December 31, 2019.
results according to Strengthening the Reporting of Obser- The Food Patterns Equivalents Database contains
vational Studies in Epidemiology reporting guidelines for
observational studies [28].
Box 2 Aspects of the analysis that varied across
analytic specifications
2.1. Analytic specifications
We used a published systematic review of observational
(1) Type of nutrition model
studies that addressed the effect of red meat on all-cause
mortality to identify justifiable analytic specifications for - Standard model
specification curve analysis [29]. We focus only on obser- - Multivariable nutrient density model
vational studies because randomized trials typically involve
the preparation of detailed protocols and statistical analysis (2) Operationalization of red meat
plans that reduce the analytic decisions available to inves-
tigators. While our objective was to investigate the effects - Continuous (per 100 g/day)
of unprocessed red meat, we did not anticipate that studies - Quartiles
- Quintiles
investigating the effects of mixed unprocessed and pro-
cessed red meat or unspecified types of red meat would
use different analytic methods. Hence, we also reviewed (3) Subgroups of interest
studies that reported on mixed unprocessed and processed
- All participants
red meat and unspecified types of red meat.
Two reviewers, working independently and in duplicate, - Subgroup based on sex
reviewed the primary studies from the systematic review
- All females
and collected data on study characteristics and analytic - All males
methods, including the type of analytic model (eg, Cox pro- - Both sexes
portional hazards model, logistic regression), method of
adjustment for energy (eg, standard model, multivariable - Subgroups based on age
nutrient density model), covariates included in the model,
operationalization of covariates (eg, categorical, linear, - Participants aged 20e39 years
quadratic), subgroup analyses (eg, men vs women), and - Participants aged 40e59 years
- Participants aged 60e79 years
the results of analyses, including secondary and sensitivity
- All ages
analyses, when reported [29]. To ensure that the primary
studies that we used to inform our analytic specifications
addressed similar causal questions and interpreted their (4) Covariates
findings similarly, we documented the objectives of the pri-
mary studies and the ways in which the authors interpreted
their findings.
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 5

information on the composition and nutritional content of interest (ie, only males, only females, all sexes, 20e39 years
individual foods. old, 40e59 years old, 60e79 years old, all ages), and choice
We acknowledge that NHANES data are likely subopti- of covariates. The standard nutrition model adjusts for total en-
mal compared to other nutrition datasets for investigating ergy in the analytic model, while the multivariable nutrient
the effect of red meat and other nutritional exposures on density model divides food intake by total energy intake and
health outcomes, due to it including few deaths and only also includes total energy intake in the model [34]. We did
collecting data on diet at a single point in time [30,32]. not consider the residual energy model since it is largely equiv-
Our objective, however, is not to provide answers about alent to the standard model [34].
the health effects of red meat but to demonstrate a proof- We constructed two sets of covariates: covariates that we
of-concept application of specification curve analysis to included in all models and covariates that were adjusted in
nutritional epidemiology. We used NHANES data due to some models. In all models, we adjusted for a core set of
its availability to our team and our team’s familiarity with covariates that were considered in nearly all primary
its structure. studies: age, sex, smoking, total energy intake, year, meno-
We observed that nearly all primary studies excluded pausal status, hormone therapy, parity, and oral contracep-
participants with missing data and performed complete case tives. We also optionally adjusted for a secondary set of
analysis. We applied the same approach and excluded par- other covariates that were only adjusted in some (but not
ticipants with missing demographic, dietary, or lifestyle in- all) studies: race/ethnicity (Mexican American/other
formation. Furthermore, we excluded pregnant people since Hispanic/non-Hispanic White/non-Hispanic Black/other
they were not included in any of the primary studies. We raceeincluding multiracial), education (less than 9th
also excluded participants with implausible body mass in- grade/9e11th grade/high school graduate/some college or
dex (BMI) (!15 or 60 kg/m2) or energy intake (! AA degree/college graduate or above), marital status,
500 kcal/day or O4500 kcal/day) since these likely repre- alcohol consumption, physical activity, BMI, socioeco-
sent instances of inaccurate reporting or collection of data. nomic status, comorbidities, and dietary variables.
To minimize missing data, we consolidated related vari- We are unable to test for all possible combinations of co-
ables in the database (eg, when data were missing for the variates due to computational feasibility. Hence, we gener-
smoking history variable, we classified participants who ated 20 random unique combinations of covariates that all
endorsed smoking 0 cigarettes in their life as nonsmokers). adjusted for the core set of variables and each of which
Participants in NHANES completed two 24-hour dietary adjusted for a random set of the secondary covariates. We
recalls, each conducted by trained interviewers and sepa- applied specification curve analysis and computed hazard
rated by 3e10 days, for which they provided information ratios (HRs) and 95% confidence intervals corresponding
on intake of foods and beverages on each recall day [32]. to the effect of red meat intake on all-cause mortality for
For our analysis, we define unprocessed red meat as any each analytic specification.
mammalian meat (ie, beef, veal, pork, lamb, and game For specifications in which red meat was treated as a
meat) [33]. continuous variable, we calculated HRs and associated con-
fidence intervals corresponding to a 100 g/day increase in
intake of red meat. For specifications in which red meat
2.3. Data analysis
was treated as a categorical variable (eg, quartiles or quin-
We performed specification curve analysis to investigate tiles), we calculated HRs and associated confidence inter-
the effects of unprocessed red meat on all-cause mortality, vals corresponding to the highest vs lowest quantile of
using a Cox proportional hazards regression model with red meat exposure. While these contrasts represent different
time since 24-hour recalls as the time variable in the model. quantities of red meat intake, primary observational nutri-
For each aspect of the analysis, we used the most used tional epidemiology studies overlook these differences
analytic choices from previous studies (Box 2) and enumer- when interpreting results and systematic reviews and
ated all combinations of these choices (within the context meta-analyses often combine these estimates from studies
of the analytic choices that we had selected for consider- using disparate quantities [25]. In our supplement, we pre-
ation in the specification curve analysis) to produce a sent results stratified by how red meat is defined in analytic
comprehensive list of all plausible and reasonable analytic models (ie, quartiles, quintiles, or continuous 100 g/day).
methods. We reviewed analytic specifications to confirm To test whether models from the specification curve
that every combination of analytic choices implemented analysis met the proportional hazards assumption, we
in the specification was indeed justifiable. Although we in- selected a sample of all specifications at random and tested
tended to exclude specifications comprised of combinations the correlation between Schoenfeld residuals and ranked
that were not defensible, we found no such cases. failure time.
Aspects of the analysis that varied across primary studies We excluded results from models that yielded what we
included the type of nutrition model (ie, standard model and considered to be implausible effect estimates (ie, studies
multivariable nutrient density model), operationalization of that yielded implausibly wide confidence intervals with
red meat (ie, continuous, quartiles, quintiles), subgroups of lower bound HR  0.2 or upper bound HR  5). A review
6 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

of analytic specifications that yielded results outside of this 3. Results

range suggested sparse data bias, where there are too few
3.1. Study characteristics
events in certain combinations of explanatory variables re-
sulting in overestimation or underestimation of effect esti- A systematic review identified 15 publications reporting
mates [35]. While these thresholds are arbitrary, they on 24 cohort studies that examined the effect of red meat on
pragmatically excluded specifications that yielded what all-cause mortality [29] (Supplement Table 1).
we considered to be results beyond the range of effects To ensure that these primary studies addressed similar
we would plausibly expect from diet and nutrition on health causal questions and interpreted their findings similarly,
outcomes. we documented the objectives of the primary studies and
We performed three statistical tests to address (1) the ways in which the authors interpreted their findings
whether the median effect estimate across all specifications (Supplement Table 2). The primary aim of all except two
is more extreme than would be expected if red meat had no of these studies was to investigate the effects of red meat
effect on all-cause mortality, (2) the proportion of specifica- on all-cause mortality. One study investigated the effects
tions that produced statistically significant effects is more of substituting total and different types of dietary protein
extreme than would be expected if red meat had no effect for carbohydrates on mortality but also presented models
on all-cause mortality, and (3) whether Stouffer’s averaged investigating the effects of isocaloric substitutions of carbo-
Z value across all specifications is more extreme than hydrates for red meat on mortality [37]. The second study
would be expected if red meat had no effect on all-cause investigated the effects of components of a traditional Sami
mortality [26]. To perform these tests, we permuted red diet, including red meat, on mortality [38].
meat intake and sampled with replacement across all partic- Studies reported 70 unique methods to investigate the ef-
ipants to yield 500 bootstrapped samples to which we fect of red meat on all-cause mortality (Supplement
applied specification curve analysis. Based on the results Tables 2 and 3). Studies varied in their choice of analytic
of the specification curve analysis to the permuted datasets, model (eg, Cox proportional hazards model, Poisson
we calculated P values using the percentage of bootstrap regression), adjustment for energy (eg, standard model,
sample with results as or more extreme than the observed multivariable nutrient density model), covariates included
results. We used an alpha of 0.05 to indicate statistical in the model, operationalizations of variables (eg, func-
significance. tional form in the model), and subgroups. Typical studies
We performed all analyses in R (Vienna, Austria; version performed time-dependent Cox regression models in which
4.1.2), using the specr package for specification curve anal- red meat was treated as a categorical variable in quartiles or
ysis [36]. Data from NHANES are publicly accessible and quintiles and adjusted for age, sex, smoking, alcohol intake,
the code to produce the results in this paper is available on physical activity, and BMI.
a public repository: https://github.com/Yumin-Wang/Red- Studies reported relative effect estimates of red meat on
Meat-Consumption—All-Cause-Mortality. all-cause mortality ranging between 0.63 and 2.31 (median:

Fig. 1. Selection of study participants from the National Health and Nutrition Examination Survey (NHANES) for inclusion in the analysis.
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 7

1.14; interquartile range [IQR]: 1.02e1.23). Supplement specification curve analysis). The analytic methods varied
Figure 1 presents the results of the analyses reported in according to the method of adjustment for energy (standard
studies. model, multivariable nutrient density model), the operation-
alization of red meat in the model (quintile, quartile, and
3.2. Participant characteristics continuous), subgroup based on sex (both sexes, male, fe-
male), subgroup based on age (all ages, 60e79 years old,
We used data from NHANES 2007 to 2014 and 40e59 years old, 20e39 years old), and covariates. Each
excluded participants without mortality data and missing model was adjusted for a core set of mandatory variables
or implausible data, leaving 10,661 eligible participants. and a random subset of 47 optional variables. Based on
Fig. 1 presents the selection of participants in the analysis. these variations in analytic choices, we calculated a total
Table 1 and Supplement Table 5 present participant of 10 quadrillion possible unique analyses.
characteristics. Our study included participants ranging Since we were unable to consider all possible unique an-
from young adults to the elderly, with approximately equal alyses, we restricted the number of combinations of covari-
representation of men and women. Most participants were ates we considered in the specification curve analysis. We
White, nonsmokers or light smokers, with a median intake generated 20 random unique combinations of covariates
of unprocessed red meat less than half a serving per day. that all adjusted for the core set of variables and each of
which adjusted for a random set of the secondary covari-
3.3. Specification curve analysis ates. This yielded a total of 1440 unique analytic specifica-
tions. These 1440 analytic specifications represent a
Using all analytic choices identified in the primary random subset of all 10 quadrillion possible analyses. We
studies, we enumerated all the ways in which the data reviewed the 1440 specifications to confirm that every com-
may be reasonably analyzed (within the context of the an- bination of analytic choices implemented in the specifica-
alytic choices that we had selected for consideration in the tion curve analysis was indeed justifiable. Although we
Table 1. Participant characteristics intended to exclude specifications comprised of combina-
Total participants, N 10,661
tions that were not defensible, we found no such cases.
We were able to accommodate most analytic choices re-
All-cause mortality, n (%) 1022 (10)
ported in primary studies using data from NHANES
Follow-up (months) 99 (65, 143)
(Supplement Tables 2 and 3). We were unable to implement
Age (years) 50 (27, 71) time-varying variables due to the cross-sectional nature of
Sex the NHANES data.
Female, n (%) 5150 (48) We implemented 1440 reasonable specifications and
Male, n (%) 5511 (52) identified 1208 unique specifications with plausible results
Dietary intakes and 232 with implausibly wide confidence intervals (lower
Unprocessed red meat (g/d) 29.5 (0, 120.2) bound HR 0.2 or upper bound HR 5). These implausible
Total energy intake (kcal/d) 1945 (1168, 3099) specifications occurred in analyses of subgroups of the total
Years of entering cohort study population that included many adjusting covariates,
2007e2008, n (%) 2311 (22) suggesting sparse data bias [35].
2009e2010, n (%) 2358 (22) Fig. 2 presents the results of the specification curve anal-
2011e2012, n (%) 2857 (27) ysis. Our specification curve analysis produced a median
2013e2014, n (%) 3135 (29) HR of 0.94 (IQR: 0.83e1.05) for the effect of red meat
on all-cause mortality. HRs ranged from 0.51 to 1.75. Of
Race/Ethnicity
Mexican American, n (%) 1321 (12)
all specifications, 435 (36.0%) yielded HRs equal to or
more than 1.0 and 773 (64.0%) less than 1.0.
Other Hispanic, n (%) 988 (9)
Of all specifications, 48 (3.97%) were statistically sig-
Non-Hispanic White, n (%) 5193 (49)
nificant. Of 48 statistically significant results, 40 had
Non-Hispanic Black, n (%) 2235 (21)
indicated red meat to reduce all-cause mortality and eight
Other Race e Including Multiracial, n 924 (9)
indicated red meat to increase all-cause mortality.
(%)
Among statistically significant effects suggesting benefit,
Smoking
we observed a median HR of 0.65 (IQR: 0.58e0.69) and,
Nonsmoker or light smoker, n (%) 8373 (79)
among statistically significant effects suggesting harm,
Moderate smoker, n (%) 437 (4)
we observed a median HR of 1.22 (IQR: 1.19e1.27).
Heavy smoker, n (%) 1851 (17) We found 45% (542/1208) of all specifications to yield
BMI (kg/m2) 28.4 (21.9, 38.5) point estimates ranging between HR of 0.90 and 1.10.
Abbreviation: BMI, body mass index.
Visual inspection of the specification curve plot suggests
Data presented as numbers and proportions or as medians (10th subgroup by sex to importantly influence results, with ana-
percentile, 90th percentile). lyses restricted to women more likely to suggest red meat is
8 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

beneficial. We observed a median HR of 1.05 (IQR: of unprocessed red meat on all-cause mortality [26]. To
0.89e1.12) for men and 0.85 (IQR: 0.77e0.93) for women. mitigate the subjectivity involved in selecting analytic
We did not identify other analytic characteristics as specifications, we sourced analytic approaches from the
consequential. literature [29]. We performed 1208 unique analyses and
Supplement Figure 2 presents the results of the specifi- found considerable variability in results, with HRs
cation curve analysis stratified by how red meat is defined ranging from 0.51 to 1.75. Our results suggest that find-
in analytic models (ie, quartiles, quintiles, or continuous ings in nutritional epidemiology studies may be contin-
100 g/day). Supplement Tables 6 to 10 and Supplement gent on analytic methods.
Figures 3 to 7 show the results of tests for the proportional In contrast to previous studies addressing red meat, we
hazards assumption and graphical displays of the correla- found few of our analytic specifications to yield statistically
tion between Schoenfeld residuals and ranked failure time. significant effects. This may be because we used more
We did not find evidence that the proportional hazards recent data from NHANES, which include fewer accumu-
assumption was violated in any analyses. lated deaths [39]. The most recent iterations of NHANES,
Finally, we present statistical inferences about the de- however, are likely more reflective of the effects of red
gree to which findings across all specifications are incon- meat on all-cause mortality in the context of contempora-
sistent with the null hypothesis (ie, red meat has no effect neous diets and lifestyles. Nevertheless, our primary objec-
on all-cause mortality) (Table 2). We performed statistical tive was not to draw inferences about the health effects of
tests addressing whether (1) the median effect estimate red meat but to provide a proof-of-concept illustration of
across all specifications, (2) the proportion of specifica- the application of specification curve analysis to nutritional
tions that produced statistically significant effects, and epidemiology.
(3) Stouffer’s averaged Z value across all specifications Concerns may arise over the impact of various analytic
is more extreme than would be expected if red meat had techniques on the interpretation of results. For example,
no effect on all-cause mortality. All three statistical tests different methods for energy adjustment may have different
yielded P values O.05. implications for how the effect is interpreted [18,40]. In our
study, we show that despite differences in analytic methods,
authors stated similar objectives and similarly interpreted
4. Discussion their results. This suggests that authors are using disparate
analytic methods to investigate near identical causal
4.1. Main findings questions.
In this study, we applied specification curve In addition to analytic flexibility, researchers criticize
analysisda method that involves defining and imple- observational nutritional epidemiology studies for biases
menting all plausible and valid analytic approaches for associated with self-reported dietary data [20,23]. Yet,
addressing a research questiondto estimate the effect nutritional epidemiology studies continue to play a critical

Fig. 2. Results of specification curve analysis. (For interpretation of the references to color in this figure legend, the reader is referred to the Web
version of this article.)
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 9

Table 2. Inferential statistics

P value
(% of bootstrap sample with
Test statistics used Observed results results as or more extreme)
Median effect size HR 5 0.94 P 5 .472
Share of significant results 48 of 1208 specifications P 5 .998
Aggregate all P values Stouffer Z 5 11.69 P 5 .732

Abbreviation: HR, hazard ratio.

role in shaping dietary recommendations and policies [15]. Similarly, since there is usually more than one dataset avail-
While specification curve analysis does not address biases able to address the same research question, the choice of
due to dietary measures, when combined with other tools dataset is also a subjective decision. As specification curve
and methods for more reliably measuring diet, specification analysis becomes more common in epidemiology, we
curve analysis may have the potential to enhance confi- expect more of such subjective factors to emerge. Nonethe-
dence in the discipline [41,42]. less, specification curve analysis does improve on current
practice in which investigators can test many alternative an-
alytic specifications and selectively report results for those
4.2. Relation to previous work
that yield interesting or favorable results. It can identify
Current evidence shows that results from studies may findings that are most robust to alternative analytic specifi-
vary due to alternative analytic specifications and that there cations and encourage evidence users to interpret the results
is often limited consensus on the optimal approach for data of epidemiology studies considering the typical variation in
analysis [6,43]. Research to date has not, however, quanti- results expected due to analytic flexibility.
fied the magnitude of variation in results for typical epide- Specification curve analysis also does not eliminate the
miologic questions. Furthermore, while specification curve need for content knowledge and expertise. We see content
analysis has been previously applied in psychology and expertise being essential to distinguishing between justifi-
economics, it has not yet been applied in epidemiology or able and nonjustifiable analytic specifications and interpret-
nutritional epidemiology [27,44,45]. ing and contextualizing results. In this study, content
expertise in nutrition was critical to select methods to adjust
for energy, the choice of core variables that we included in
4.3. Strengths and limitations
all analytic models, and the interpretation of our findings.
The current work offers an innovative solution to ana- We did not register a protocol for the present study. This
lytic flexibility in nutritional epidemiology. To our knowl- study is intended to provide a proof-of-concept rather than
edge, our work is the first application of specification test any specific hypotheses. Since our work presents the
curve analysis to nutritional epidemiology. first or one of the first applications of specification curve
Our study also has limitations. There may be disagree- analysis to epidemiology, we expected to encounter many
ments among investigators about what constitutes a justifi- unanticipated decisions and challenges that we could not
able analytic approach. To mitigate this issue, our choice of predict or describe in a protocol. Hence, our work was
analytic specifications was informed by primary studies and largely exploratory. The repository containing the analytic
so represents real, published analyses rather than possible, code also contains a history of the project from its inception
unpublished analyses that may only be marginally defen- in 2021.
sible. Ideally, investigators should prespecify criteria for Different analytic methods may have implications for
distinguishing between justifiable and unjustifiable analytic the interpretation of results. For example, different methods
approaches. We caution against investigators in making to adjust for energy intake in nutritional epidemiology
these distinctions after implementing the analysis since address different causal questions [18]. Authors of nutri-
their decisions may be influenced by the observed results. tional epidemiology studies, however, seldom acknowledge
We emphasize that specification curve analysis does not these issues. We show that despite differences in analytic
eliminate subjectivity. For example, investigators may methods, authors stated similar objectives and similarly in-
disagree about what constitutes a justifiable analytic terpreted their results.
approach. Furthermore, if investigators choose to select an- We only applied specification curve analysis to one
alytic specifications based on published literature, as we did questiondthe effect of red meat on all-cause mortality.
in this study, there is typically more than one published sys- The extent to which results may be contingent on analytic
tematic review that can be used to identify primary studies methods may be different for other questions. We acknowl-
and the choice of systematic review may be subjective. edge that this is a controversial question in the nutrition
10 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

literature and that the application of specification curve threshold that had too few events to reliably estimate the ef-
analysis to less contentious questions in nutritional epide- fect of red meat on all-cause mortality.
miology may improve its adoption. Our choice of topic Finally, while we attempted to test the proportional haz-
was influenced by our team’s familiarity with red meat ards assumption using the correlation between Schoenfeld
and the related literature [15,29]. residuals and ranked failure time, these tests have limited
Our study likely underestimates the variations in results sensitivity [49]. We also only tested a proportion of our
due to alternative analytic specifications since the analytic models for the proportional hazards assumption and it is
specifications that we could implement were limited by possible that this assumption may be violated in models
the availability of variables and data in NHANES. For that we did not test.
example, due to the cross-sectional nature of NHANES,
we were unable to use time-varying covariates and explore
4.4. Implications
how alternative ways to account for these variables may in-
fluence results. We did not account for potential subjec- Specification curve analysis allows investigators to test
tivity in inclusion of participants in the analytic set (eg, all plausible and justifiable models to explain conflicting
thresholds for extreme energy intake) to maintain similar findings or contextualize emerging findings. While this
numbers of participants across analyses. Similarly, there study may provide insights on the health effects of unpro-
are subjective analytic decisions in translating dietary re- cessed red meat, we believe the most important contribu-
calls to nutrient and food intake, although we could not ac- tion of this study is to provide a proof-of-concept
count for these decisions. For example, nutritional demonstrating the feasibility of applying specification
epidemiologists code dietary recalls according to food clas- curve analysis to nutritional epidemiology.
sification systems and subsequently use nutrition databases Nutritional epidemiology has long been criticized for pro-
to estimate individual nutrient components of each item in ducing sensational and conflicting findings, which has eroded
dietary recallsdall of which involves subjective decisions. confidence in the discipline [23]. Nevertheless, nutritional
The continuous 2007e2014 NHANES data are likely epidemiology studies continue to play a crucial role in shaping
suboptimal for investigating the effect of red meat and other dietary recommendations and policies, making it imperative
nutritional exposures on health outcomes, due to it to draw credible inferences from these studies [14,15,24].
including few deaths and only collecting data on diet at a The broader application of specification curve analysis to
single point in time [30,32]. Nevertheless, our primary nutritional epidemiology may enhance confidence in nutrition
objective is not to provide conclusive answers about the as a field by encouraging investigators to acknowledge an
health effects of red meat but to demonstrate a proof-of- additional source of uncertainty in studies. When combined
concept application of specification curve analysis to nutri- with other tools and methods that also address other limita-
tional epidemiology. tions of observational nutritional epidemiology studies (eg,
We did not incorporate weights in our analytic models. biases that affect self-reported dietary data) [41], specification
Sample weights in NHANES are designed to account for curve analysis has the potential to address a critical issue in
oversampling of specific subgroups and unequal probabil- epidemiologydanalytic flexibilitydand identify findings
ities of selection in the population. These weights are that are most robust to subjective analytic choices.
essential when the objective is to make inferences about Findings from our study and future application of spec-
population characteristics or to estimate prevalence rates ification curve analysis will also be useful to evidence users
because they adjust for factors that influence these esti- who can interpret results of epidemiology studies in the
mates and ensure that the results are representative of context of the typical variation expected due to analytic
the target population. However, when focusing on causal flexibility. When effect estimates exceed the typical varia-
inference, the primary concern is to eliminate or control tion due to analytic methods, evidence users can be more
for confounding factors that may distort the true relation- certain of the findings, since they are likely robust to alter-
ship between exposure and outcome and sample weights native analytic decisions.
are less important, especially when variables used to Our findings may also have implications for precision
derive sample weights are already included in analytic nutrition that attempts to distinguish between subgroups
models [46e48]. of individuals who may differently respond to nutritional
We excluded models that yielded results that we deemed interventions or have different nutritional needs [50e52].
to be implausible based on pragmatic but arbitrary thresh- Investigators have raised concerns that efforts to identify
olds (ie, HR 0.2 or HR 5). We suspect that the observed ‘‘responders’’ and realize precision nutrition may be highly
implausible specifications were due to sparse data biasd dependent on the characteristics of analytic models [53].
where there are too few events in critical combinations of Specification curve analysis may be useful for evaluating
explanatory variables [35]. It is, however, possible that the reliability of precision nutrition claims across a range
there were other models that produced results within this of defensible models.
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 11

We acknowledge that the application of specification The lower part of the plot shows the characteristics of
curve analysis is time-consuming and resource-intensive. each analysis, including type of analytic model, operation-
Sourcing justifiable analytic specifications from primary alizations of variables, choice of covariates, and subgroups
studies adds to this effort. While the application of speci- of interest. Each vertical line denotes the specific choice
fication curve analysis may not be feasible for all nutri- applied for each aspect of the analysis. We assigned a
tional epidemiology questions, it can be applied to the unique number to each covariate (Supplement 4 shows
most critical, impactful, or contentious questions in the the number corresponding to each variable). Combinations
discipline and can serve as an additional available tool of numbers in the graph represent combinations of covari-
to evaluate the credibility of nutrition claims in the ates included in the model.
literature.
This is one of the first applications of specification curve
analysis to epidemiologic health data. We anticipate further Ethics approval
refinement of the method with future applications,
including the development of more comprehensive guid- Not required.
ance for investigators and increased standardization of the
approach. For example, ideally, investigators should
prespecify how they will select analytic aspects to consider Patient/public engagement
in specification curve analyses and how they will distin- It was not possible to involve patients or the public in the
guish between justifiable and unjustifiable analytic ap- design, conduct, reporting, or dissemination plans of our
proaches. Furthermore, the interpretation of results from research.
specification curve analysis is currently complex. Specifica-
tion curve plots may be overwhelming for evidence users,
especially if they account for many different analytic as- CRediT authorship contribution statement
pects. We hope with the greater adoption of this method,
improved ways of communicating results from specifica- Yumin Wang: Writing e review & editing, Writing e
tion curve analyses emerge. original draft, Validation, Formal analysis, Data curation,
Conceptualization. Tyler Pitre: Writing e review & edit-
ing, Writing e original draft, Methodology, Investigation,
Formal analysis. Joshua D. Wallach: Writing e review
5. Conclusion & editing, Writing e original draft, Methodology, Data cu-
In this study, we apply specification curve analysisda ration, Conceptualization. Russell J. de Souza: Writing e
novel analytic method that involves defining and imple- review & editing, Writing e original draft, Methodology,
menting all plausible and valid analytic approaches for ad- Investigation, Conceptualization. Tanvir Jassal: Writing
dressing a research questiondto investigate the effect of e review & editing, Writing e original draft, Project
red meat on all-cause mortality. We show variability in re- administration, Methodology, Investigation. Dennis Bier:
sults across plausible analytic specifications. This research Writing e review & editing, Writing e original draft,
demonstrates how specification curve analysis can be effec- Methodology, Investigation, Formal analysis. Chirag J.
tively applied to nutritional epidemiology, providing a prac- Patel: Writing e review & editing, Writing e original
tical and innovative solution to analytic flexibility. This draft, Validation, Supervision, Project administration,
approach has the potential to improve the credibility of in- Methodology, Funding acquisition, Formal analysis. Dena
ferences from such epidemiologic studies. Zeraatkar: Writing e review & editing, Writing e orig-
This figure presents the results of the specification curve inal draft, Visualization, Supervision, Resources, Project
analysis, including 1208 unique analytic specifications. The administration, Methodology, Investigation, Formal anal-
upper portion of the plot shows HRs representing the effect ysis, Data curation, Conceptualization.
of red meat on all-cause mortality. On the x-axis are the
unique analytic specifications. The y-axis represents the
magnitude of effect estimates. Each point on the graph rep- Data availability
resents the results of a unique analytic specification. Point Data will be made available on request.
estimates are shown in dark gray and 95% confidence inter-
vals as light gray bars. Each point represents the results for
the effect of red meat on all-cause mortality for a unique
Declaration of competing interest
model. Points in blue are statistically significant and sug-
gest red meat to prevent all-cause mortality and points in The authors declare that they have no known competing
red are statistically significant and indicate red meat to in- financial interests or personal relationships that could have
crease risk of all-cause mortality. appeared to influence the work reported in this paper.
12 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278

Acknowledgments [17] Willett WC, Stampfer M, Tobias DK. Re: adjustment for energy
intake in nutritional research: a causal inference perspective. Am J
None. Clin Nutr 2022;116:608e9.
[18] Tomova GD, Arnold KF, Gilthorpe MS, Tennant PWG. Adjustment
for energy intake in nutritional research: a causal inference perspec-
tive. Am J Clin Nutr 2022;115:189e98.
Supplementary data [19] Wallach JD, Serghiou S, Chu L, Egilman AC, Vasiliou V, Ross JS,
et al. Evaluation of confounding in epidemiologic studies assessing
Supplementary data related to this article can be found at alcohol consumption on the risk of ischemic heart disease. BMC
https://doi.org/10.1016/j.jclinepi.2024.111278. Med Res Methodol 2020;20:64.
[20] Schoenfeld JD, Ioannidis JP. Is everything we eat associated with
cancer? A systematic cookbook review. Am J Clin Nutr 2013;97:
References 127e34.
[21] Gkiouras K, Choleva ME, Verrou A, Goulis DG, Bogdanos DP,
[1] Tierney BT, Anderson E, Tan Y, Claypool K, Tangirala S, Kostic AD, Grammatikopoulou MG. A meta-epidemiological study of posi-
et al. Leveraging vibration of effects analysis for robust discovery in tive results in clinical nutrition research: the good, the bad
observational biomedical data science. PLoS Biol 2021;19(9): and the ugly of statistically significant findings. Nutrients
e3001398. 2022;14(23):5164.
[2] Patel CJ, Burford B, Ioannidis JP. Assessment of vibration of effects [22] Hall KD. Challenges of human nutrition research. Science 2020;367:
due to model specification can demonstrate the instability of observa- 1298e300.
tional associations. J Clin Epidemiol 2015;68:1046e58. [23] Ioannidis JPA. Unreformed nutritional epidemiology: a lamp post in
[3] Chu L, Ioannidis JPA, Egilman AC, Vasiliou V, Ross JS, Wallach JD. the dark forest. Eur J Epidemiol 2019;34(4):327e31.
Vibration of effects in epidemiologic studies of alcohol consumption [24] Ley SH, Ardisson Korat AV, Sun Q, Tobias DK, Zhang C, Qi L, et al.
and breast cancer risk. Int J Epidemiol 2020;49:608e18. Contribution of the nurses’ health studies to uncovering risk factors
[4] Hoogeveen S, Sarafoglou A, Aczel B, Aditya Y, Alayan AJ, Allen PJ, for type 2 diabetes: diet, lifestyle, biomarkers, and genetics. Am J
et al. A many-analysts approach to the relation between religiosity Public Health 2016;106:1624e30.
and well-being. Religion Brain Behav 2022;13:1e47. [25] Zeraatkar D, Bhasin A, Morassut RE, Churchill I, Gupta A,
[5] Breznau N, Rinke EM, Wuttke A, Nguyen HHV, Adem M, Lawson DO, et al. Characteristics and quality of systematic reviews
Adriaans J, et al. Observing many researchers using the same data and meta-analyses of observational nutritional epidemiology: a
and hypothesis reveals a hidden universe of uncertainty. Proc Natl cross-sectional study. Am J Clin Nutr 2021;113:1578e92.
Acad Sci U S A 2022;119:e2203150119. [26] Simonsohn U, Simmons JP, Nelson LD. Specification curve analysis.
[6] Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, Nat Human Behav 2020;4(11):1208e14.
et al. Many analysts, one data set: making transparent how variations [27] Rohrer JM, Egloff B, Schmukle SC. Probing birth-order effects on
in analytic choices affect results. Adv Methods Pract Psychol Sci narrow traits using specification-curve analysis. Psychol Sci 2017;
2018;1(3):337e56. 28(12):1821e32.
[7] Madigan D, Ryan PB, Schuemie M. Does design matter? Systematic [28] von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC,
evaluation of the impact of analytical choices on effect estimates in Vandenbroucke JP. The Strengthening the Reporting of Observational
observational studies. Ther Adv Drug Saf 2013;4(2):53e62. Studies in Epidemiology (STROBE) statement: guidelines for report-
[8] Zeraatkar D, Cheung K, Milio K, Zworth M, Gupta A, Bhasin A, ing observational studies. Lancet 2007;370:1453e7.
et al. Methods for the selection of covariates in nutritional epidemi- [29] Zeraatkar D, Guyatt GH, Alonso-Coello P, Bala MM, Rabassa M,
ology studies: a meta-epidemiological review. Curr Dev Nutr 2019; Han MA, et al. Red and processed meat consumption and risk for
3(10):nzz104. all-cause mortality and cardiometabolic outcomes. Ann Intern Med
[9] van Dongen NNN, van Doorn JB, Gronau QF, van Ravenzwaaij D, 2020;172:511e2.
Hoekstra R, Haucke MN, et al. Multiple perspectives on inference [30] Centers for Disease Control and Prevention. National center for
for two simple statistical scenarios. Am Stat 2019;73(sup1):328e39. health statistics. National health and nutrition examination survey
[10] Landy JF, Jia ML, Ding IL, Viganola D, Tierney W, Dreber A, et al. data 2020. Available at: https://wwwn.cdc.gov/nchs/nhanes/
Crowdsourcing hypothesis tests: making transparent how design continuousnhanes/default.aspx. Accessed October 3, 2022.
choices shape research results. Psychol Bull 2020;146(5):451e79. [31] Centers for Disease Control and Prevention. National center for
[11] Schilling KG, Rheault F, Petit L, Hansen CB, Nath V, Yeh F-C, et al. health statistics. NDI mortality data. centers for disease control and
Tractography dissection variability: what happens when 42 groups prevention 2020. Available at: https://www.cdc.gov/nchs/data-
dissect 14 white matter bundles on the same dataset? Neuroimage linkage/mortality.htm. Accessed October 3, 2022.
2021;243:118502. [32] Ahluwalia N, Dwyer J, Terry A, Moshfegh A, Johnson C. Update
[12] Low J, Ross JS, Ritchie JD, Gross CP, Lehman R, Lin H, et al. Com- on NHANES dietary data: focus on collection, release, analytical
parison of two independent systematic reviews of trials of recombi- considerations, and uses to inform public policy. Adv Nutr 2016;
nant human bone morphogenetic protein-2 (rhBMP-2): the Yale 7(1):121e34.
Open Data Access Medtronic Project. Syst Rev 2017;6(1):28. [33] Wiseman M. The second World Cancer Research Fund/American
[13] Scientific Pandemic Influenza Group on Modelling. SPI-M-O: Institute for Cancer Research expert report. Food, nutrition, physical
consensus statement on COVID-19, 8 October 2020. 2020. Available activity, and the prevention of cancer: a global perspective. Proc Nutr
at: https://www.gov.uk/government/publications/spi-m-o-consensus- Soc 2008;67(3):253e6.
statement-on-covid-19-2-february-2022. Accessed March 2, 2024. [34] Willett WC, Howe GR, Kushi LH. Adjustment for total energy intake
[14] Ruxton C. Interpretation of observational studies: the good, the bad in epidemiologic studies. Am J Clin Nutr 1997;65:1220Se8S. discus-
and the sensational. Proc Nutr Soc 2022;81(4):279e87. sion 9S-31S.
[15] Zeraatkar D, Johnston BC, Guyatt G. Evidence collection and evalu- [35] Greenland S, Mansournia MA, Altman DG. Sparse data bias: a prob-
ation for the development of dietary guidelines and public policy on lem hiding in plain sight. BMJ 2016;352:i1981.
nutrition. Annu Rev Nutr 2019;39:227e47. [36] Masur P, Scharkow M. ‘‘specr: conducting and visualizing specifica-
[16] Willett W. Nutritional epidemiology. New York, NY: Oxford Univer- tion curve analyses (Version 1.0.0).’’. 2020. Available at: https://
sity Press; 2012. CRAN.R-project.org/package5specr. Accessed February 8, 2023.
 Y. Wang et al. / Journal of Clinical Epidemiology 168 (2024) 111278 13

[37] Kelemen LE, Kushi LH, Jacobs DR Jr, Cerhan JR. Associations [44] Orben A, Przybylski AK. The association between adolescent well-being
of dietary protein with disease and mortality in a prospective and digital technology use. Nat Human Behav 2019;3(2):173.
study of postmenopausal women. Am J Epidemiol 2005;161: [45] Carter EC, Sch€onbrodt FD, Gervais WM, Hilgard J. Correcting for
239e49. bias in psychology: a comparison of meta-analytic methods. Adv
[38] Nilsson LM, Winkvist A, Brustad M, Jansson JH, Johansson I, Methods Pract Psychol Sci 2019;2(2):115e44.
Lenner P, et al. A traditional Sami diet score as a determinant of mor- [46] Winship C, Radbill L. Sampling weights and regression analysis. So-
tality in a general northern Swedish population. Int J Circumpolar ciol Methods Res 1994;23(2):230e57.
Health 2012;71(0):1e12. [47] Andrew G. Struggles with survey weighting and regression modeling.
[39] Kappeler R, Eichholzer M, Rohrmann S. Meat consumption and Stat Sci 2007;22(2):153e64.
diet quality and mortality in NHANES III. Eur J Clin Nutr 2013; [48] Solon G, Haider SJ, Wooldridge J. What Are We Weighting For?
67(6):598e606. Cambridge, MA: National Bureau of Economic Research Working
[40] Tomova GD, Gilthorpe MS, Tennant PW. Theory and performance of Paper Series; 2013:18859.
substitution models for estimating relative causal effects in nutri- [49] Stensrud MJ, Hernan MA. Why test for proportional hazards? JAMA
tional epidemiology. Am J Clin Nutr 2022;116:1379e88. 2020;323:1401e2.
[41] Kirkpatrick SI, Baranowski T, Subar AF, Tooze JA, Frongillo EA. [50] Kirk D, Catal C, Tekinerdogan B. Precision nutrition: a systematic
Best practices for conducting and interpreting studies to validate literature review. Comput Biol Med 2021;133:104365.
self-report dietary assessment methods. J Acad Nutr Diet 2019; [51] Rodgers GP, Collins FS. Precision nutrition-the answer to "what to
119(11):1801e16. eat to stay healthy". JAMA 2020;324:735e6.
[42] Subar AF, Freedman LS, Tooze JA, Kirkpatrick SI, Boushey C, [52] Bailey RL, Stover PJ. Precision nutrition: the hype is exceeding the
Neuhouser ML, et al. Addressing current criticism regarding science and evidentiary standards needed to inform public health rec-
the value of self-report dietary data. J Nutr 2015;145(12): ommendations for prevention of chronic disease. Annu Rev Nutr
2639e45. 2023;43:385e407.
[43] Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing trans- [53] Fr€ohlich H, Balling R, Beerenwinkel N, Kohlbacher O, Kumar S,
parency through a multiverse analysis. Perspect Psychol Sci 2016; Lengauer T, et al. From hype to reality: data science enabling person-
11(5):702e12. alized medicine. BMC Med 2018;16(1):150.