Module 4: Participant Sampling

Learning Outcomes
- Understand how a study's sampling procedure impacts the generalizability of its results
- Understand the concept of random sampling errorf
- Explain how the procedure that is used to collect a sample and the sample size impact
the magnitude of random sampling error
- Explain the role of a sample frame in the sampling process
- Recognize and distinguish different approaches to sampling, including simple random
sampling, stratified sampling, convenience sampling, and snowball sampling
- Use a research randomizer to generate random numbers that can be used for a
sampling task
- Recognize when a particular sampling procedure should be used for a given sampling
task
- Think critically and discuss concerns about possible cultural biases in psychological
studies that rely on convenience samples

Sampling and Generalization: Challenges and Practical Solutions

- Sampling: the process of selecting individuals to participate in a study

- Population: all members of the category of interest
- Sample: subset of the population who the researcher selects as participants
- Population parameters: attributes of the population whose value the researcher is trying
to estimate
- Sample statistic: values of the attributes that the researcher observed in sample (ex.
average number of symptoms of stress reported by the undergrads who participated in
the survey)
Sampling and Generalization
- The logic that we can base generalizations on observations of samples is something that
we should be familiar with from many situations in everyday life
- Ex. a cook can taste just a bit of the meal to get the jist of the whole dish
- Sample is used to estimate what something is generally like
- Specifically, psychological researchers base generalizations about people’s minds and
behaviour on their observations of the responses of selected samples
- Ex. suppose that a psychologist wants to study the
prevalence of symptoms of stress among
undergraduates at the University of Waterloo
- The parameter that is being estimated is the
prevalence of symptoms of stress within the
undergraduate population. In 2016/2017 there
were 29,997 registered full-time undergraduates at
UW. Because the researcher does not have the
time and resources to survey all of these people
instead she will recruit a smaller sample of 1,500
 students to participate in her survey. The average number of symptoms in the
sample is the sample statistic that will be used to estimate the average number of
symptoms in the population
- The quality of the procedures that are used to recruit a sample determines the accuracy
with which that sample’s observed responses can be used to estimate the corresponding
population parameters
- The study’s sampling methods are thus relevant to assessing its external
validity
- If researchers use careful methods to ensure that the characteristics of
the participants in their sample will reflect the characteristics of the target
population then this will help enhance the external validity of the study
and bolster the researchers’ confidence that they can generalize the
results to the broader population
- However, if there are biases or other flaws in a study's sampling
method then the sample may not be representative of the
population and the external validity of the study's estimates will be
questionable
- There’s risks that samples may be inaccurate or even biased representations of the
source that they were drawn from leads researchers to take careful measures to try and
ensure that their samples will be accurate representations of the phenomena that they
are studying
- Everyday life example. Laughing at a comedy trailer only to be disappointed by
the actual movie

A Step-by-Step Guide to Sampling

- Coverage error: failure to include members of the population of interest within the
sampling frame
- Multi-stage sampling: type of probability sampling that involves conducting random
sampling in a sequence of stages at multiple levels of organization of the population
- Probability sampling: sampling procedure where the probability of any given case being
selected can be determined and the cases are sampled independently of each other
- Representative sample: a sample whose properties accurately reflect those of the target
population
- Sampling frame: defined sense of the population that identifies population members for
sampling (ex. list of all individuals in the target population)
- Simple random sampling: every individual listed in the sample frame has equal
probability of being selected for inclusion in the sample
Collecting Samples
- Sampling is a multi-step process, and researchers need to be mindful of the choices that
they make for each of these steps in the process
1. Identify target population to which you intend to generalize the result of your study
- Depending on research question, could be fairly narrow or broader population
2. Define a sampling frame that identifies the members of the population and that you can
use to draw your study from
- A sampling frame is usually not a perfectly comprehensive listing of the
population
- Ex. if population is Ontario adults and you choose to use the Ontario
telephone directories as your sampling frame - this is not a convenient
source since not all adults have phones
- These exclusions are referred to as coverage error
- Ultimately, the results of a study can only be generalized to the population that
falls within the sampling frame, not to the cases that were excluded from the
sampling frame
- So if telephone directories were used, the results of your study would only
be able to be generalized to the narrower population of individuals listed
in Ontario telephone directories rather than the broader population that
you intended to generalize your results to
3. Collect a representative sample (select individuals within sampling frame to recruit
participants in your study)
- A sample whose properties accurately reflect those of the target population
- To achieve a representative sample researchers strive to use probability
sampling techniques
- A sampling procedure qualifies as probability sampling if it fulfills the following
criteria:
- the probability (or likelihood) that a given member of the population will be
selected into the sample is a known quantity (e.g., the researcher is able
to say that the chance that any given case will be sampled is something
like 1 in 50, or 1 in 126, or 1 in 500, etc.), and
- the cases are selected independently of each other, meaning that the
selection of any given case into the sample does not change the
probability that any other case will be sampled
- One type of probability sampling technique is simple random sampling (SRS)
- Every individual who is listed in the sample frame has an equal probability
of being selected for inclusion in the sample
- Ex. lottery system in which the name of each member of the
population is listed on a separate paper and mixed
- In SRS the samples are usually drawn without replacement, which
means that an individual can only be drawn once for inclusion in the study
- Researchers usually use a computerized random number generator to produce a
table of random numbers that they can use to select cases from their sample
frame
- To do this, researched first must determine the size of the sample (n) that
they intend to select from the sample frame (N)
- Then, they assign number from 1 to N to each individuals listed in their
sample frame
 - Next, they use a random number generator to generate a list
of random numbers and select the matching number until
they reach their target sample size (n)
- Simple random sampling is not always a practical sampling method
to implement especially if the target population is widely dispersed
- Ex. if researcher seeked to randomly select Ontario citizens
for in-person interview - there are 414 towns/cities in
Ontario, with simple random sampling the researcher would
likely have to travel to most of these towns and cities to
conduct these interviews - could be very inefficient and
research might not have the resources and personnel to
accomplish this
- Multi-stage sampling involves conducting random sampling in a
sequence of stages across multiple levels of organization of the population
- In example where researcher is seeking to collect a representative
sample of Ontario citizens to interview, they could begin by random
selecting 40 towns within Ontario
- Next, they might randomly select 10 street blocks within each of the
selected towns
- Finally, they might randomly select 5 households on each
of the selected blocks
- This way a researcher would be able to sample 2000
Ontarians, but they would only need to send their
research personnel to 40 separate towns to conduct
these interviews (rather than potentially having to visit
400+ towns) - because random sampling was
implemented at each stage of selection, this should help
to promote representativeness of the sample
Think and Respond

1. Did this researcher use simple random sampling?

- No. This is not simple random sampling because every name does not have an equal
chance of being included in the sample. Names that are listed first and last have 100%
 chance of being selected while names in the middle of the page have 0% chance of
being selected.
2. Think about how this sampling bias could distort the results of the study. Describe one
way that selecting the first and last names on each page might cause the characteristics
of the sample to differ from those of the population.
- The last name on a page and the first name on the next page are very close to each
other alphabetically. So, sampling the first and last names on every page will lead the
researcher to oversample individuals who happen to have the same name and thus
might be related (e.g., parent and child, siblings, cousins). When these individuals are
interviewed they might report that having a family connection to someone else in the
profession was one of the critical factors that drew them to a career in the law. This is
likely to overestimate the importance of family connections in the law profession because
lawyers with family connections were unintentionally over-sampled.

Stratified Random Sample

- Strata: subgroups of the population who share some characteristic in common (ex.
Generational cohorts within a population - Baby Boomers, GenXers, Millennials)
- Stratified random sampling: type of probability sampling in which the researcher
randomly samples within specified strata
Stratified Random Sampling
- In many studies a researcher seeks to compare the responses of defined segments of
the population called strata
- Ex. a researcher studying stress in the UW population might intend to compare
the prevalence of stress among students in each of the different faculties ex.
Arts, AHS, Engineering etc.
- If SRS was used, her final sample might not include enough participants
from some of the smaller faculties to allow for meaningful comparisons
with the larger faculties
- To ensure the sample will contain sufficient numbers of participants to
permit meaningful comparisons of these groups the researcher can use
stratified random sampling instead of simple random sampling
- Stratified random sampling: the researcher first reorganizes the sample frame to
identify cases that belong to each of the specific groups that she wishes to
compare - researcher would take the list of registered students and segment it by
faculty, separating out the names on the list into each of the separate faculties
- Researcher would then choose a random sample of participants from
within each of these faculty lists (ex. Randomly select n cases from the
Arts faculty, and n cases from the AHS faculty etc.)
Illustration of Stratified Random Sampling
- Imagine that you had a list of 420 UW students who participated in a volunteer program
- You want to study their satisfaction with their experience in the program and you would
like to be able to compare the average satisfaction levels of students from different
faculties
- To enable these comparisons you plan to interview 20 of these students from each of the
UW faculties

1. Reorganize table into separate lists for each of the faculties (the strata in your stratified
sample)

2. Assign the individuals in each list an identifying number

- Usually numbered sequentially assigning 1 to the first name in the list etc. until you reach
last name

- Now have 6 lists that you can randomly sample from

- To generate the random numbers that you will use to select these cases you can use a
computerized random number generator
- To illustrate how this works we will use a Research Randomizer tool - this
randomizer is a web-tool that the Social Psychology Network (SPN) offers as a
free service for researchers around the world
- SPN’s randomizer is an award-winning tool that has been cited in many
published research articles [https://www.randomizer.org/]
Using the Research Randomizer Tool
1. The first question asks how many sets of numbers (lists) you want to generate
- In the end we need to generate 6 lists of random numbers
- However, it is better to generate these lists sequentially rather than all at once
because we have a different number of cases in each list and one of the later
steps requires you to enter the range of numbers that you need to select from.
So, for now we'll ask for just 1 set which we will use to select cases from the Arts
list
2. The second question asks how many numbers you need per set
- In our example since we are seeking to sample 20 individuals from each facult
we would enter 20
3. The third question asks for the number range
- Since there are a total of 126 cases in our Arts list we should enter a range from
1 to 126
4. We want each number in the set to remain unique, so select “yes” to the next question
- This generates a number from the specified range only once per list generated
- This allows the researcher to sample without replacement, which means that a
given case in the sample frame can only be selected once
- Sampling without replacement is the default approach in most psychological
studies
5. Sorting the numbers that are generated makes it easier to find and select your
participants from your list. Select ‘yes, least to greatest’ to the fifth question
6. The sixth question asks if you would like place markers. These provide sequential
enumeration beside each random number generated. This can be left off, “Place
Markers Off”
7. Once the set of parameters are entered, click “Randomize Now!” and the tool will
generate a list of random numbers, which can be used to select participants to create a
random sample

Random Sampling Error

- Random sampling error: discrepancies between the sample statistics and population
parameters that are due to random, or chance-based, differences between the sample
and population
- Variance: the amount of variability in some measured quantity
When the Sample Statistic Does Not Reflect the Population
- In everyday life we have experiences where a sample of observations leads to a
misleading impression just due to chance
- Random sampling error refers to any discrepancies between the sample statistics and
the population parameters that occur due to such chance factors
- Random sampling is the most reliable approach for collecting a representative target
population
- However, because the sample is just a subset of the population there is some
likelihood that the characteristics of the sample will differ from the characteristics
 of the population just based on change even if the participants were selected
through randomly sampling
- Ex. imagine that there’s a class of 20 students all taking the same class
- 15 (75%) have a favourable opinion of the course and 5 (25%)
have an unfavourable opinion
- If you randomly draw the names of 4 students to fill out the course
evaluation
- By chance 2 of the sampled students happen to come from the
group with a favourable opinion and the other 2 come from the
group with an unfavourable opinion
- Thus, in the sample the favourable to unfavourable ratio is
1:1 Whereas the actual
population ratio is 3:1
- The more variability there is in the population on the
attribute that is being measured, the higher the random
sampling error will tend to be
- Fortunately, there’s a fairly straightforward way
to lower the magnitude of random sampling
error - simply increase your sample size
- All else being equal, you will have a larger
random sampling error if you draw a small
sample from a population than you would have
if you drew a larger sample from that population
- *a relatively large sample will give you a more reliable
estimate of your target population parameter than a
relatively small sample would give you
- Researchers can use what they know about the
population’s general heterogeneity to estimate how
much variability there is likely to be in the
characteristics they are attempting to measure in order
to plan how large a sample they will need to address
random sampling error
- Ex. if a group of researchers is studying a highly
heterogeneous population that has extensive diversity in socioeconomic status,
ethnicity, age, religious background etc. then they might anticipate that there will
be relatively high variability in the psychological characteristics that they
measure, and thus they would plan to recruit a relatively large sample
- If the same study was conducted in a very homogenous population whose
members tend to be quite similar in their backgrounds and circumstances,
then the researchers might anticipate relatively low variability in the
characteristics - therefore a smaller sample size can be made
Sample Size and Variance Interactive Demonstration
- Sample size and variance in an outcome measure or observation, such as stress level,
can impact how reliable a statistic is of the degree to which we can trust results of the
study

- Red curve: the mean (average score) and variance (distribution of scores) within the
population from which we are sampling
- Blue bars: reflect the number or frequency of individuals in the sample and their score
- If our sample happens to have a lot of variance in it then we’ll need to use a
much larger sample size in order to get a reliable estimate that approximates the
true population mean
- Ex. we are measuring the prevalence of stress in 2 populations of undergraduates - we
will consider drawing a sample from 2 populations that have the same average level of
stress (mean , µ) of '13', as measured using a perceived stress scale
- Population A has high variance in stress scores, meaning that the scores of the
individuals in this population differ quite a bit from each other and from the population
mean with many cases of individuals that have stress scores that are considerably lower
and relatively few individuals whose scores are considerably higher than the population
mean
- Ex. this population might consist of students at different levels of study (first
through fourth year students) and in many different majors across all of the
different faculties at the university - because these majors differ quite a bit in the
levels of stress and competitive pressures that they place on students and
because stress levels might differ quite a bit depending on a student’s study
term, we would expect there to be quite a wide range of variability in how
stressed these students are
- Population B has low variance in stress scores, meaning that the scores of the
individuals in this population do not differ as much from each other and from the
population mean, and there are relatively few individuals that have stress scores that are
considerable lower and many others that are considerably higher than the population
mean
- Ex. this population might consist of students who are all in the same year of study
(second year students only) and all registered in the same major - because the
major and level of study are the same for all these students there might be a lot
less variability in how stressed these students are
- Now we will draw a series of relatively small and relatively large samples from each
population to see how the variance in the population and the sample size influence the
reliability of the sample estimates
- Starting with population A:
1. Set the population variance to be high to reflect the fact that this population has
high variance in stress scores
2. Set the sample size to be low. Set the sample size to draw 25 cases from this
population
- Write down the sample mean that you get after you draw your first sample
of 25 cases [12.90107]
- Select ‘resample’ 5 more times to draw more samples of 25 cases from
this population and again write down the sample means. You should now
have 6 sample means
- 6 sample means: 12.90107, 12.56030, 12.48131, 13.26972, 12.81068,
13.16858
3. Compare the 6 samples means that you got when you drew these 6 small
samples from Population A. How much do they differ from each other and how
much do they differ from the actual population mean of 13?
4. Now let’s see what happens when you increase your sample size. Set the
sample size to draw 250 cases from this population
- Write down the sample mean that you get after you draw your first sample
of 250 cases [13.07251]
- Select ‘resample’ 5 more times to draw more samples of 250 cases from
this population and again write down the sample means. You should now
have 6 sample means from this larger sample
- 6 samples: [13.07251, 13.13019, 13.02103, 12.91974, 13.05259,
12.86772]
5. Compare the 6 sample means that you got using the large samples from
Population A. How much do they differ from each other and how much do they
differ from the actual population mean?
6. How do your results differ when using smaller versus larger samples?
- In the graph, the blue bars are closer to the red line when there’s bigger
sample size
- Now let’s follow the same procedure with Population B
 1. Set the population variance to be low to reflect the fact that this population has
low variance in stress scores
2. Follow steps 2-6 from above but sampling from this lower variance population

Think and Respond

What you may have noticed is that when the variance in our sample is high and our sample size
is small the mean estimates vary quite a bit from each other and the true population mean.
Often in psychology we do not know the true population mean, so we approximate it by trying to
sample in a way that reduces variance due to sampling error or noise and we try to use large
enough sample sizes that we can trust our estimates of the population mean.

Systematic Sampling Error

- Systematic sampling error: discrepancies between the sample estimate and the
population value that occur when certain members of the population are less likely to be
included in the sample compared to other members of the population
- Selective nonresponse: a type of systematic sampling error that arises when certain
members of the population are less available or less motivated to participate in a study
and thus are underrepresented in the sample
When the Distribution of Members of the Sample Do Not Reflect the Population
- While random sampling error is a pretty straightforward problem to deal with, the other
major type of sampling error - systematic sampling error - poses more serious
challenges
- It occurs when certain members of the population are more likely to be included in the
sample than other members
- Systematic oversampling or undersampling of certain members of the population can
lead to a distorted estimate of the population parameter if the variable that influences the
likelihood of being samples is related to the variable(s) that the researcher is trying to
estimate
- Ex. researchers who are trying to estimate the prevalence of stress in the
population of Waterloo students will systematically underestimate this value if
something about their recruitment procedure causes highly stressed students to
less likely to choose to participate in the survey compared to students who are
less stressed
- Suppose highly stressed students feel like they don’t have time to devote
to the survey because they’re feeling pressured about their regular
coursework - if highly stressed students are more likely to opt out of
participating in the survey then they will be undersamples relative to their
population
- The undersampling of the most stressed students will mean that the
prevalence of stress that is recorded in the sample will be lower than the
actual prevalence of stress in the undergrad population
- Even if researcher randomly selects representatives from
the population, there are other factors that might introduce
systematic bias into the sample
- Selective nonresponse: individuals typically need to
consent to participate in a study, if the individuals who
choose to participate differ systematically from those who
opt out then this will cause the sample to be
unrepresentative of the population
- Ex. if in an election poll the supporters of one of the
candidates are predisposed to distrust the pollster then they would be
more likely to refuse to participate and consequently the poll may
underestimate the prevalence of support for this candidate in the
population
- in the 2016 US Presidential election many commentators
speculated that pre-election polls might have underestimated
support for Donald Trump because his supporters may have been
more likely to refuse to participate in the polls compared to Hillary
Clinton's supporters perhaps because Trump's voters distrust the
mainstream news media (e.g., CNN, The New York Times) that
sponsor these polls
- Systematic sampling errors due to factors like selective nonresponse are less
easy to correct for than random sampling error
 - Opposite of random sampling error, collecting a larger sample size will
only influence the researchers to be more confident about a biased
estimate
- Researchers should be careful to examine their samples to see if there is
evidence of systematic sampling error such as a low overall response rate or
patterns indicating that the individuals who refuse to participate differ on some
systematic basis (ex. Higher response rates in more affluent neighbourhoods
compared to less affluent neighbourhoods)
- Some ways researchers can avoid systematic sampling errors such as selective
nonresponse:
- Adding incentives to participate, sending an advance letter to inform
potential participants when and why they are being contacted for the
study, or adjusting the recruitment process to ensure that participants with
a variety of backgrounds and interests will feel welcome to participate and
will be motivated to take part in the study
- When there is evidence of systematic error in a sample such as selective
nonresponse of certain demographic groups then researchers may attempt to
make certain statistical adjustments to correct for these sample biases such as
applying higher weights to the responses of underrepresented groups. If these
statistical corrections are applied carefully they can help to mitigate systematic
sampling biases
Biased Sampling
Think and Respond

- You may have noticed that many of your social network contacts are similar to you in
political views, ethnicity, age, religious beliefs, educational background, musical tastes,
sexual preferences, and many other ways. For example, if you're a secular humanist
then it's likely that your social contacts tend to be more secular than the broader
population. You thus might be relatively underexposed to the ideas and opinions of
religious people. This might lead you to underestimate the role that religion plays in
many people's lives.

- Make efforts to expose yourself to information outside your personal network, especially
information sources that might be on the opposing side of most of your contacts. For
example, if you're a liberal and most of your friends and acquaintances are also liberals,
then make an effort to visit websites that have a conservative point-of-view to see how
conservatives might be interpreting some of the issues and news topics that you and
your friends are discussing or vice versa if you are conservative.
 - If most of your contacts have a similar opinion on an issue but a few of your contacts
have an opposite opinion on that issue then make extra efforts to engage with the
contacts who have this minority opinion and ask them to direct you to other sources of
information on the topic when it comes up in conversation. This way you can leverage
contacts who are different from you to expose yourself to a more diverse range of
opinions and ideas, enabling you to make more informed decisions and conclusions.

Sampling Strategies When There Is No Available Sampling Frame

- Systematic random sampling: selecting cases for observation according to a pre-defined,

sequenced schedule
- Ad lib sampling: sampling at the researcher’s whim (ex. In an arbitrary, undefined way)
- Confirmation bias: the tendency to seek out or interpret evidence in ways that are biased
to favour the researcher’s hypothesis
What to Do When there is No Sampling Frame
- In some cases it may not be possible to construct a sampling frame for the population -
in such cases no list of all members of the population exists, or the members of the
population may not be able to be tracked for the duration of the study
- In these cases, researchers may use a systematic random sampling strategy to avoid
biases in sampling
- Ex. suppose that a psychologist hypothesizes that teenage girls tend to be more
emotionally expressive when they are in public spaces than teenage boys tend to
be
- To test this hypothesis, the researcher needs to observe teenagers’
behaviours in a public setting such as a mall to construct a sample frame
and then randomly sample them
- It also would not be feasible to record all of the teenagers’ behaviour - so
some sampling process is necessary, but simple random sampling would
not work
- Ad lib sampling: just recording the behaviour of as many teenage boys
and girls as he can during the period of observation
- Ad libbing is not recommended because there are a number of
biases that might influence which individuals catch the
researchers’ attention
- Ex. the researcher might be unintentionally biased to
notice the examples that match the researcher’s
predictions and overlook or discount examples that did not
match their hypothesis (confirmation bias)
Think and Respond
If the confirmation bias influences the researcher's sampling choices then this would
compromise the internal validity of the study. The confirmation bias would compromise the
study's internal validity because if the collected data show that teenage girls are more
emotionally expressive than teenage boys we would not know whether this was because there
is an actual relationship between gender and emotional expressivity or because the researcher
oversampled cases that supported their hypothesis. In other words, the researcher's sampling
bias would be an alternative explanation of any findings in this study. As we discussed in
Module 3, when an alternative explanation of a finding such as the influence of a confirmation
bias cannot be ruled out this weakens a study's internal validity. This example shows that
sampling biases not only may compromise a study's external validity by undermining the
researcher's ability to generalize the results to the target population, as we've been emphasizing
in this module, but sampling biases might also compromise a study's internal validity by
introducing alternative explanations of any patterns of results obtained.
- To avoid the biases associated with ad lib sampling researchers in situations like this
should instead use systematic random sampling
- In systematic random sampling the researcher predetermines how many cases
they plan to observe - they then use a random number generator to select 2
number - one to determine when they will begin sampling and another to
determine the subsequent schedule for sampling
- In mall example, if a random number generator selects ‘7’ and ‘4’ then the
researcher would begin recording the behaviour of the 7th teenager they see and
then record the behaviour of every 4th teenager after that until they reach their
target sample size
- Eliminates the influence of researcher biases because the research doesn’t personally
choose which cases to observe, rather the random number generator selects the cases
- Systematic random sampling is useful in many observational studies where researchers
are recording the behaviour of individuals or groups in some field setting (ex. a
comparative psychologist observing the mating behaviour of a bird species; a
developmental psychologist observing interactions of parents and children at a mall)
- there is usually not an opportunity to randomly sample cases from a membership
list but the researcher can use systematic random sampling to select cases for
observation in a way that will not be contaminated by researcher biases such as
the confirmation bias

Nonprobability Samples

- Nonprobability samples: Samples for which the researcher cannot determine the
probability that various members of the population are included in the sample.
- Convenience sampling: Samples that are recruited based on the researcher's
convenience rather than based on their representativeness of the population.
- Location-dependent sampling: Sampling that relies on the clustering of members of the
target population at accessible physical locations or online forums.
- Snowball sampling: Sampling that uses social network contacts of an initial set of
participants to recruit other members of a target population.
 - Homogeneity bias: Reliable tendency of people to affiliate with others who are similar to
them in a number of sociological and psychological features such as demographic
characteristics, personality traits, social identity characteristics, beliefs, and attitudes.
- Triangulation strategy: This is the strategy of using a variety of methods or approaches
to recruiting participants in a study (e.g., sampling from different locations) in an effort to
counteract sampling biases associated with any particular recruitment method.
A Representative Sample is Not Always Possible
- Although probability samples such as random samples are ideal for the purpose of
generalizing the results of a study to a target population, in many cases researchers may
be unable to collect a representative sample
- These cases are called nonprobability samples because the researcher does not know
the probability that particular cases in the population will be selected into the sample
- A common type of nonprobability sampling is convenience sampling
- A convenience sample is constructed based on the accessibility of its members
to the researcher rather than based on the probability that its members’
characteristics will reflect those of the broader population
- Often used in experimental research where the goal is to test some hypothesis
about causal mechanisms, not to estimate the characteristics of some population
of interest - Typically in experimental research there is an assumption that the
processes studied are so fundamental that they can be generalized beyond the
relatively narrow samples of convenience that are typically used
- Representative samples are most critical when a researcher is trying to form a reliable
estimate of some population value
- Ex. in political surveys a representative sample of voters’ candidate preferences
is needed to derive a reliable estimate of which candidate will win on election day
- Some psychological studies do aim to estimate population values
- Ex. A clinical psychologist may seek to estimate the prevalence of various
psychological disorders in some communities. In cases such as this a
representative sample would be necessary to construct a reliable estimate
- However, most psychological studies do not seek to estimate specific population values
on some psychological variable. Instead, they seek to test a hypothesis about the causal
mechanisms that influence some psychological response
- Samples of convenience have low external
validity and thus are a poor basis for generalization
about population characteristics
- However, in many experimental
psychology studies there is more of a priority of
maximizing internal validity to test hypotheses about
psychological mechanism and the researchers are
less concerned about whether the results of their
research have external validity for generalizing the
results beyond the sample
- Another context where nonprobability sampling strategies might need to be used is
research that studies hidden populations
- Hidden populations are populations where there is not any existing,
comprehensive public record of the membership of the population that can be
used to form a sampling frame
- Ex. population might be hidden when its members share some socially
stigmatized characteristic that is relatively uncommon and concealable
(ex. People with opiate addictions, sexual minorities, undocumented
immigrants, political extremists)
- If a researcher seeks to study the members of some hidden population then they cannot
use the usually recommended strategy of randomly sampling cases from a membership
list. Instead, for hidden populations researchers often use a combination of
location-dependent sampling and snowball sampling
- Location-dependent sampling relies on the fact that people who share a
characteristic tend to cluster together in physical (or virtual) spaces
- Ex. Researcher who is seeking to document the psychological effects of
homophobia on LGBT individuals might seek to recruit a sample of LGBT
individuals by advertising her study through a network of LGBT support
groups
- Capitalizing on clustering of people in reliable locations can be a convenient way to find
and recruit people from hidden populations. However, one shortcoming of
location-dependent sampling is that the individuals who cluster in these accessible
locations might not be representative of that hidden population as a whole

- To mitigate some of the sampling biases associated with location-dependent sampling

researchers may employ another strategy referred to as snowball sampling
- In this strategy the members of a community who a researcher finds at a
convenient clustering location are just the starting point for recruiting participants
 - The researcher uses these individuals as her initial point of access into a hidden
population but then capitalizes on these individuals' personal social networks as
a resource for moving deeper into that population
- The way this works is that the researcher asks the participants who are recruited
through location-dependent sampling to provide her with contact information for
other members of this population of interest
- The researcher then attempts to recruit further participants from the pool of
names that were provided by the first round of participants. These secondary
participants may also be asked to provide the names of contacts to attempt to
move even further into the target population
- Goal of snowball sampling is to recruit a more diverse and inclusive sample than a
researcher might get through location-dependent sampling alone
- While snowball sampling might help to extend a researcher’s research into a
hidden population there’s no guarantee that it will be sufficient to overcome the
sampling biases associated with the initial location
- People’s social networks tend to have a well-known homogeneity bias -
meaning that people tend to affiliate with other who are similar to themselves in
many ways, including similarity in their demographics, attitudes, experiences,
personality traits, and other psychological characteristics
- Ex. if an initial sample tended to oversample younger members of a
population then many of their personal acquaintances are also likely to be
relatively young and the list of contacts that they provide the researcher
are likely to perpetuate the youth-biased skew of the initial sample
- Given the many limitations and potential for systematic biases to contaminate
nonprobability samples, researchers should be very cautious about basing any
generalizations on convenience samples, location-dependent samples, and
snowball samples
- Researchers should be aware that these sampling strategies have the potential
to give a biased representation of the target population. There are likely to be
sample-dependent biases in any estimates that are produced through these
nonprobability sampling strategies
Think and Respond
https://www.opendemocracy.net/beyondslavery/ashley-greve-oliver-kaplan/can-snowball-sampli
ng-estimate-human-trafficking

- If this method of snowball sampling would tend to recruit the more socially connected
victims of human trafficking then this would oversample the least vulnerable members of
this population. The members of the population who are more socially connected are
 relatively less hidden and thus may have access to victims' services, support networks,
and legal supports. By contrast, less socially connected members of the population are
likely to be living more restricted and controlled lives that cut them off from these
external resources. To the extent that this research method would tend to oversample
the more socially connected victims of human trafficking then it could lead researchers to
underestimate how oppressed and vulnerable many victims of human trafficking actually
may be.

Summary

- Sampling involves a series of choices that researchers make for recruiting the
participants in their studies. These choices determine how confidently they can
generalize their results beyond their sample. First, researchers choose to specify what
target population they intend to generalize their results to. Second, researchers choose
a sample frame to identify the members of their target population. The quality of the
sampling frame that participants choose determines the coverage error in their sample.
Third, researchers choose a method to select cases within their sampling frame for
participation in the study. Random selection of cases from the sampling frame is one of
the most useful methods for ensuring that the study will have a representative sample
- Random and systematic errors in the sampling process can lead the sample results to
deviate from whatever population values the researcher is estimating. Researchers can
mitigate random sampling error by increasing their sample size. Systematic sampling
error is a more challenging problem because it involves bias in either the recruitment
process or in participant responsiveness. To avoid systematic error researchers try to
monitor their sampling process to detect signs of bias and then implement measures to
eliminate or statistically adjust for these biases
- While methods such as random sampling are the preferred approach for recruiting a
representative sample of a population, psychological researchers often rely on samples
that have questionable representativeness, such as samples of convenience.
Convenience samples may be adequate for many research purposes such as in
experiments where the researcher does not aim to generalize the results to estimate a
population value with precision but instead aims to test the validity of a hypothesis about
some psychological processes. Psychological researchers may also need to rely on
other sampling methods that have questionable representativeness such as
location-based sampling and snowball sampling when they try to recruit members of
hidden populations that are composed of individuals who have rare or potentially
stigmatized characteristics. In such cases where the representativeness of the sample is
questionable, researchers will need to be cautious about basing any broader
generalizations about the characteristics of the group on the qualities that they observe
in their samples