A simple formula for the calculation of sample size in

pilot studies
Citation for published version (APA):

Viechtbauer, W., Smits, L., Kotz, D., Budé, L., Spigt, M., Serroyen, J., & Crutzen, R. (2015). A simple
formula for the calculation of sample size in pilot studies. Journal of Clinical Epidemiology, 68(11), 1375-
1379. https://doi.org/10.1016/j.jclinepi.2015.04.014

Document status and date:

Published: 01/11/2015

DOI:
10.1016/j.jclinepi.2015.04.014

Document Version:
Publisher's PDF, also known as Version of record

Document license:
Taverne

Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can
be important differences between the submitted version and the official published version of record.
People interested in the research are advised to contact the author for the final version of the publication,
or visit the DOI to the publisher's website.
• The final author version and the galley proof are versions of the publication after peer review.
• The final published version features the final layout of the paper including the volume, issue and page
numbers.
Link to publication

General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these
rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
• You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above,
please follow below link for the End User Agreement:
www.umlib.nl/taverne-license

Take down policy

If you believe that this document breaches copyright please contact us at:
repository@maastrichtuniversity.nl
providing details and we will investigate your claim.

Download date: 11 ago. 2022

 Journal of Clinical Epidemiology 68 (2015) 1375e1379

A simple formula for the calculation of sample size in pilot studies

Wolfgang Viechtbauera, Luc Smitsb, Daniel Kotzc,d, Luc Bud
ee, Mark Spigtc,f,
Jan Serroyeng, Rik Crutzenh,*
a
Department of Psychiatry and Psychology, MHeNS School for Mental Health and Neuroscience, Maastricht University, PO Box 616,
6200 MD Maastricht, The Netherlands
b
Department of Epidemiology, CAPHRI School for Public Health and Primary Care, Maastricht University, PO Box 616, 6200 MD Maastricht,
The Netherlands
c
Department of Family Medicine, CAPHRI School for Public Health and Primary Care, Maastricht University, PO Box 616, 6200 MD Maastricht,
The Netherlands
d
Institute of General Practice, Medical Faculty, Heinrich-Heine-University D€usseldorf, D€usseldorf, Germany
e
Department of Midwifery Education and Studies, Research Centre for Midwifery Science, Zuyd University, PO Box 1256, 6201 BG Maastricht,
The Netherlands
f
Department of Community Medicine, General Practice Research Unit, University of Tromsø, Tromsø, Norway
g
Department of Methodology and Statistics, CAPHRI School for Public Health and Primary Care, Maastricht University, PO Box 616, 6200 MD Maastricht,
The Netherlands
h
Department of Health Promotion, CAPHRI School for Public Health and Primary Care, Maastricht University, PO Box 616, 6200 MD Maastricht,
The Netherlands
Accepted 29 April 2015; Published online 6 June 2015

Abstract
One of the goals of a pilot study is to identify unforeseen problems, such as ambiguous inclusion or exclusion criteria or misinterpre-
tations of questionnaire items. Although sample size calculation methods for pilot studies have been proposed, none of them are directed at
the goal of problem detection. In this article, we present a simple formula to calculate the sample size needed to be able to identify, with a
chosen level of confidence, problems that may arise with a given probability. If a problem exists with 5% probability in a potential study
participant, the problem will almost certainly be identified (with 95% confidence) in a pilot study including 59 participants. Ó 2015
Elsevier Inc. All rights reserved.

Keywords: Pilot study; Sample size; Problem detection; Rule of three; Unforeseen problems

1. Introduction smaller scale can help to identify unforeseen problems that

could compromise the quality or flow of the study [2].
A pilot study can be defined as a small-scale study that
For example, one may encounter nonanticipated reasons
helps to examine the practicality and feasibility of the
why potential participants have to be excluded, question-
methods to be used in a subsequent larger and more naire items that are interpreted in unintended ways by the
comprehensive investigation [1]. Because conducting an participants or whose answer options are not sufficiently
adequately powered study often requires the inclusion of
comprehensive, or unclear information about the delivery
a large number of participants and therefore may be very
of the intervention (eg, dosing or visiting schedules).
costly in terms of time and money, piloting a study on a
If such problems are discovered during the course of a
pilot study, the necessary steps can be taken before the actual
large-scale study is started to minimize or entirely avoid their
W.V. derived the equation. W.V. wrote the first draft and produced the negative impact. For example, the study protocol or materials
figure. All authors contributed to revising the article and approved the final (eg, questionnaire items) could be adapted accordingly, or
version. W.V. is the guarantor. contingency plans could be set up ahead of time to handle
Conflict of interest/Financial disclosure: Neither conflict of interest nor
financial support regarding the present study.
any problems adequately and in a timely manner. However,
* Corresponding author. Tel.: þ31 43 388 28 28 this is only possible if such problems are actually discovered
E-mail address: rik.crutzen@maastrichtuniversity.nl (R. Crutzen). during the conduct of the pilot study. Therefore, a pilot study
http://dx.doi.org/10.1016/j.jclinepi.2015.04.014
0895-4356/Ó 2015 Elsevier Inc. All rights reserved.
1376 W. Viechtbauer et al. / Journal of Clinical Epidemiology 68 (2015) 1375e1379

if it is really lower (so that we are more likely than desired

What is new? to miss the problem), then it would be infrequent enough to
not be considered a problem worthy of detection. Choice of
Key findings this value therefore depends on the context and on how
 A simple formula to calculate the sample size detrimental a problem would be to a trial. We will return
needed to be able to identify, with a chosen level to this issue further in the following.
of confidence, problems that may arise with a
given probability in a pilot study.
2. Required sample size to detect a problem in a pilot
What this study adds to what was known? study
 Although sample size calculation methods for pilot
studies have been proposed, none of them are For now, assume that a particular problem has a given
directed at the goal of problem detection. probability of occurring in a potential study participant.
For example, if there is a 0:15 probability of encountering
What is the implication and what should change unanticipated reasons for exclusion in a given participant,
now? then there is 0:85 probability that this problem does not
 The equation can be easily adopted as a method for manifest itself. In a group of n participants, there is then
sample size calculations in pilot studies that is sim- a 0:85n probability that the problem will not occur at all.
ple to use, but provides a basis for more reasoned Therefore, the probability of observing at least one
decisions about sample sizes in pilot studies. occurrence of this problem in n participants is given by
PðxO0Þ 5 1  ð1  pÞ ;
n

where x denotes the number of participants (of the n

aimed at discovering such problems should have sufficient participants) in whom the problem manifests itself and p
power to do so, or in other words, the sample size of a pilot denotes the problem probability. We now want to choose
study must be sufficiently large, such that the probability of n so that PðxO0Þ exceeds a certain threshold of confidence,
detecting such problems is high. which we denote by 100%  g. In other words, how many
Existing methods for sample size calculations typically participants must be included so that we can be 100%  g
focus on how to select an appropriate sample size for a pilot certain that the problem will manifest itself at least once
study such that various parameters of interest can be during our pilot study? Solving the equation for n yields:
estimated with sufficient precision (eg, the effect size, the lnð1  gÞ
standard deviation of the outcome measure, its reliability, n5 : ð1Þ
lnð1  pÞ
or adherence or attrition rates) [1,3e5]. Such calculations
may also play an important role in deciding whether to Because n will typically not be a whole number, we
proceed with the primary trial in the first place [6,7]. These round the value obtained with Equation (1) up to the nearest
considerations have led to various guidelines for choosing integer. An online calculator is available at http://www.
an appropriate sample size for a pilot study, such as 12 pilotsamplesize.com.
participants per group [3], values in the range of 10 to 40 For example, for p50:15 , including n5ln (1  0.95)/
participants per group depending on the parameter of ln(1  0.15 )5 18.43, or rather 19 participants will ensure that
interest [4,5], at least 9% of the main trial’s sample size we will encounter at least one incident of the problem at a 95%
[6], or at least 50 participants [8]. confidence level. A problem that manifests itself less frequently,
However, none of these approaches is directly applicable for example, with only p50:05 probability, will require the
when the goal of a pilot study was the detection of inclusion of lnð1  0:95Þ=lnð1  0:05Þ 558:40 or rather 59
unforeseen problems. Therefore, in this article, we take a participants in the pilot study so that the problem can be
different approach and describe a simple method for detected with a high confidence level.
determining the sample size necessary to identify problems Fig. 1 shows the required sample size (ie, n) as a func-
with a chosen level of confidence in pilot studies. tion of the problem probability (ie, p) for three different
Not surprisingly, the sample size determined in this levels of confidence (ie, g equal to 0.95, 0.90, and 0.80).
manner depends not only on the confidence level with Not surprisingly, higher confidence levels and the detection
which we would like to detect a particular problem but also of less-frequent problems require larger sample sizes.
the actual probability that the problem manifests itself in a
potential study participant. Because the true problem
probability is unknown in practice, what we really need 3. Sample size calculations in practice
to consider is a lower bound for the problem probability:
if the true probability is in fact this low (or higher), then To use Equation (1) for deciding on a sample size for a
we achieve (or exceed) the desired confidence level, but pilot study, we must first choose values for g and p. By
 W. Viechtbauer et al. / Journal of Clinical Epidemiology 68 (2015) 1375e1379 1377

If the true probability is actually higher than p50:10,

then the achieved confidence level will exceed 95% when
29 participants are screened (eg, approximately 99% if
p50:15, as one can easily verify with Equation 1). There-
fore, screening 29 participants will ensure a high level of
confidence (ie, at least 95%) for the chosen minimum prob-
lem probability. On the other hand, if the actual problem
probability is lower than p50:10, then 29 participants will
not be sufficient to reach the desired 95% confidence level.
However, in that case, because the true problem probability
is actually lower than the minimum level deemed to be
important for detection, we can also consider it acceptable
that we run a higher risk of ‘‘not’’ encountering at least one
case of ambiguity (because the investigator can easily
handle a smaller number of ‘‘on the spot’’ decisions).
Note that the ‘‘minimum problem probability’’ is
Fig. 1. Required sample size to detect a problem with a given level of
context dependent. For example, suppose the presence or
confidence. absence of a particular condition constitutes the primary
outcome variable in a trial. Diagnoses are made indepen-
dently by two clinicians, and disagreements would demand
convention, we could adopt a confidence level of 95% additional follow-up assessments. If such assessments are
(analogous to the level commonly chosen for confidence costly and/or invasive, we may already find a disagreement
intervals) or choose the confidence level in accordance with probability of (at least) p50:01 to be problematic and
the severity of the issue that we want to detect. (ie, a therefore would want to make sure that this issue is likely
potentially disastrous problem that could ruin the entire to be discovered during a pilot study (eg, so that ways of
study should be detected with higher confidence than a reducing the occurrence of disagreements can be
problem that would merely be a nuisance to deal with.) considered). On the other hand, if costs and invasiveness
With respect to p, we need to consider that very infrequent are minimal, we may consider a higher probability of
problems will naturally require a large sample size to detect. p50:05 or p50:10 to be acceptable (but anything higher
However, infrequent problems (unless they are so disastrous might call into question the reliability with which diagnoses
as to require the immediate discontinuation of the study) can are made).
be handled on an as-needed basis, without disrupting or jeop- Similarly, if responses to a questionnaire are used to
ardizing the entire study. When choosing p, one should there- measure the primary outcome in a trial, then we may again
fore consider the amount of resources (eg, time and choose a lower minimum probability for detecting partic-
personnel) available to handle unforeseen difficulties and ular problems (eg, nonresponses, item misinterpretations)
their impact on the trial. In addition, because the ‘‘true’’ prob- than if the same questionnaire was part of a process evalu-
ability of a particular problem is technically unknown in ation, where problems that occur with a low probability are
practice, we suggest thinking of p as the ‘‘minimum’’ prob- not detrimental to the trial itself. However, if problems
ability of the problem that we would like to detect with the manifest themselves with at least a p50:05 or p50:10
desired confidence level. probability, we would still hope to be alerted of their
For example, suppose the inclusion or exclusion criteria existencedeven if it is not the primary outcome in the trial.
for a study have been written to the best of the researchers’ Finally, in practice, one will often be interested in not
knowledge, there may be participants for whom the criteria just a single but an entire collection of different problems
do not yield an unambiguous outcome. Whenever such a (eg, difficulties in applying the inclusion or exclusion
case presents itself, for example, during a screening visit, criteria, ambiguities in questionnaire items, problems in
a decision about the eligibility of the participant needs to the application of the treatment). Equation (1) can then
be made. Suppose that a single investigator is responsible be used to determine the required sample size to detect each
for making this decision. A large number of such cases individual problem. The largest value of n obtained this
would then quickly overburden this investigator. way then guarantees that each individual problem can be
Accordingly, it is decided that, if such difficulties present detected with ‘‘at least’’ the desired confidence level
themselves with ‘‘at least’’ p50:10 probability (ie, in at specified.
least 1 of 10 participants), it would be good to detect this For example, suppose that, in addition to potential
problem already during the pilot study. Accordingly, ambiguities with the inclusion or exclusion criteria, the
Equation (1) then indicates that 29 participants need to be researchers also want to detect any problems participants
screened to be 95% confident that one or more such cases may have with the interpretation of the items on a question-
are in fact encountered. naire. Because incorrectly interpreted items may
1378 W. Viechtbauer et al. / Journal of Clinical Epidemiology 68 (2015) 1375e1379

a probability of 0.02 could still be detected with almost

Box The rule of three 80% confidence.
It needs to be emphasized that the method does not
A method for constructing confidence intervals for
indicate the appropriate sample size for estimating the actual
p when observing zero events (ie, when x50) in a
probability of a particular problem (ie, p) with a given level
study [13,14], sometimes called the ‘‘rule of three,’’
of precision [1,5]. Instead, the method is used to determine
is closely related to Equation (1). Applied to the
the necessary sample size so that the problem is likely to be
present context, the method works as follows: If the
observed at least once during the course of the pilot study.
problem does not manifest itself in the n
The emphasis therefore is on ‘‘problem detection’’ and not
participants included in the pilot study, then an
on the estimation of the ‘‘problem frequency.’’
approximate 95% confidence interval for p is given
Similarly, it is often suggested that pilot studies are
by the interval with endpoints 0 and 3=n. For
useful for determining an appropriate sample size for
example, suppose with 60 participants, we never
large-scale investigations, for example, by providing infor-
observe the problem in the pilot study. Then, the
mation about the variability in the outcome variable of in-
endpoints of an approximate 95% confidence
terest or regarding the size of the effect. However,
interval for p are equal to 0 and 0.05. Note that
because pilot studies are by definition conducted with small
Equation (1) for p50:05 leads in fact to n559, the
sample sizes, they usually do not provide accurate informa-
difference resulting only from the approximation
tion about such parameters and therefore may lead to
used in the derivation of the confidence interval
erroneous decisions when such estimates are used without
[13]. Therefore, Equation (1) essentially implies
further qualification [2,10]. The same issue applies when
that one should include 3=p participants if one
using pilot studies to estimate other parameters of interest
wants to be approximately 95% certain that the
when planning studies (eg, adherence or attrition rates).
problem will manifest itself at least once during the
Finally, it is worth noting that Equation (1) has been
pilot study.
discussed in other contexts, such as the number of animals
required to detect at least a single case of disease with a
given confidence level during long-term holding of
completely invalidate the responses from the participants laboratory rodents [11] and the required number of objects
that encounter such difficulties, the investigators want to (typically precincts) that must be audited to detect at least
be 95% certain that they will detect this issue during the one case of election fraud [12]. In addition, Equation (1)
pilot study if the problem manifests itself with ‘‘at least’’ is related to what is sometimes called ‘‘the rule of three,’’
p50:05 probability (ie, in at least 1 of 20 participants). a method for computing confidence intervals for propor-
As described earlier, 59 participants should then be tions (see Box). As described in the present article, the
included in the pilot study. This will also ensure that any equation can be easily adopted as a method for sample size
ambiguities with the inclusion or exclusion criteria are calculations in pilot studies, which is simple to use but
detected with more than a sufficient confidence level. provides a basis for more reasoned decisions about sample
sizes in pilot studies.

4. Discussion References
Despite the extensive levels of planning involved in the [1] Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tuto-
design of a study, experience shows that unforeseen problems rial on pilot studies: the what, why and how. BMC Med Res Method-
can and typically will arise during the conduct of a study that ol 2010;10:1.
must be handled with care. A pilot study provides an excel- [2] Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution
regarding the use of pilot studies to guide power calculations for
lent opportunity to uncover such problems ahead of time, study proposals. Arch Gen Psychiatry 2006;63:484e9.
minimizing the need to adapt procedures or to develop con- [3] Julious SA. Sample size of 12 per group rule of thumb for a pilot
tingency plans on short notice when the larger study is being study. Pharm Stat 2005;4:287e91.
conducted. In this article, we described a simple method for [4] Hertzog MA. Considerations in determining sample size for pilot
studies. Res Nurs Health 2008;31:180e91.
choosing a sample size for a pilot study that ensures the dis-
[5] Johanson GA, Brooks GP. Initial scale development: sample size for
covery of potential problems with high confidence. pilot studies. Educ Psychol Meas 2010;70:394e400.
How have existing pilot studies fared in the context of [6] Cocks K, Torgerson DJ. Sample size calculations for pilot random-
these results? Arain et al [9] found a median sample size ized trials: a confidence interval approach. J Clin Epidemiol 2013;
of 76 participants in pilot studies published in 2007 or 66:197e201.
2008 in four general medical and three specialist journals. [7] Stallard N. Optimal sample sizes for phase II clinical trials and pilot
studies. Stat Med 2012;31:1031e42.
For this median sample size, problems with a probability [8] Sim J, Lewis M. The size of a pilot study for a clinical trial should be
of approximately 0.04 could have been detected with a calculated in relation to considerations of precision and efficiency. J
95% confidence level, whereas problems that occurred with Clin Epidemiol 2012;65:301e8.
 W. Viechtbauer et al. / Journal of Clinical Epidemiology 68 (2015) 1375e1379 1379

[9] Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or [12] Aslam JA, Popa RA, Rivest RL. On estimating the size and confi-
feasibility study? A review of current practice and editorial policy. dence of a statistical audit. Proceedings of the 2007 USENIX Work-
BMC Med Res Methodol 2010;10:67. shop on Accurate Electronic Voting Technology; 2007, Boston.
[10] Browne RH. On the use of a pilot sample for sample size determina- [13] Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all
tion. Stat Med 1995;14:1933e40. right? Interpreting zero numerators. JAm Med Assoc 1983;249:1743e5.
[11] Institute of Laboratory Animal Resources. Long-term holding of lab- [14] Eypasch E, Lefering R, Kum CK, Troidl H. Probability of adverse events
oratory rodents. ILAR News 1976;19:1e25. that have not yet occurred: a statistical reminder. BMJ 1995;311:619e20.