 MDM4U

Organization of Data For Analysis

Collecting Primary
Data: Effective
Surveys

MINDS ON

In 2016, the Government of Canada

commissioned Vox Pox Labs, a Canadian
statistics company, to create a survey
and report on the data on Canadian
opinions with respect to government.
The purpose of the survey was to start a
discussion on electoral reform, or a
possible change to the elections process
that the country currently has in place for
federal elections.

15 million Canadian households received

an invitation to complete the online
survey with 383,074 unique respondents.
The survey results, an extensive 134
page report, can be found at
mydemocracy.ca. Note: p. 21-25 of the
report discusses the demographic
breakdown of respondents vs. the total
population.

by CBC

Record Your
Work
Some of the questions asked of
Canadians include the following. It's
your turn. Answer each question and
provide a reason why you think
that way.

1. It is better for several parties to

have to govern together than for
one party to make all the
decisions in government, even if
it takes longer for government to
get things done.
a. Strongly disagree
b. Somewhat disagree
c. Neutral
d. Somewhat agree
e. Strongly agree

2. A party that wins the most seats

in an election should still have to
compromise with other parties,
even if it means reconsidering
some of its policies.
a. Strongly disagree
b. Somewhat disagree
c. Neutral
d. Somewhat agree
e. Strongly agree

3. Governments should have to

negotiate their policy decisions
with other parties in Parliament,
even if it is less clear who is
accountable for the resulting
policy.
a. Strongly disagree
b. Somewhat disagree
c. Neutral
d. Somewhat agree
e. Strongly agree

4. There should be parties in

Parliament that represent the
views of all Canadians, even if
some are radical or extreme.
a. Strongly disagree
b. Somewhat disagree
c. Neutral
d. Somewhat agree
e. Strongly agree

5. Which best describes your

views?
a. Canadians should have the
option to cast their ballots
online in federal elections,
even if the security or
privacy of online voting
cannot be guaranteed OR
b. Canadians should continue
to vote using paper ballots
at a polling station, even if it
is less accessible for some
voters?

Consider the
Following
Do the above questions have flaws
in them, Yes or No? Justify your
answer.

ACTION

Bias in Collecting
Data
When writing survey questions, it is
important that the true beliefs of the
survey respondents being sampled is
measured. As with all bias, it may be
impossible to fully eliminate it, you can
only work hard to reduce it. Recall,
variability is inherent in data and will
always exist in some way.

Measurement Bias
Definition: Measurement Bias exists
when there is some flaw in the
measurement tool. For surveys, it is
when the survey has been poorly
designed. There is always some margin
of error when collecting survey
responses. In a well-designed survey,
the error is random, and does not result
in an opinion being overstated or
understated. When a survey has
measurement bias, the survey responses
are either overstated or understated.

Recall the example from Activity 3:

"Since cars pollute the environment, do
you think there should be a better public
transit system in our city?" The opinion
that public transit should be improved will
be overstated by the wording of the
question. By taking out the first part of
the question, you will remove the
measurement bias. A better question to
ask is: "Do you think there should be a
better public transit system in our city?"

Measurement Bias
Questions
Three common types of questions that
contain measurement bias are leading,
loaded and double-barreled questions.

1. A leading question is a question

that influences the responses in
some way. The "Since cars pollute
the environment" part of the question
above influences responders to say
"Yes."
2. A loaded question is a question that
assumes something about the
respondents. An example would be
"What do you love about baseball?"
This question assumes that the
people being surveyed have
something they love about baseball.
The question "What do you love
about baseball?" seems very simple,
but needs to be thought through very
carefully to get true results from
respondents.
One possible solution to help
minimize the measurement bias
is to remove the word "love" to
make it easier for people to
answer. There are cultural
biases toward the word love.
You can also only ask the
question to people that said they
enjoy baseball. Online surveys
will have ways of skipping
questions for given answers.
You can also include options that
allow respondents to say "Prefer
not to answer" or "I do not enjoy
anything about baseball."
You would also want to consider
if the options you give are an
exhaustive list of everything you
can think of and then you
include "Other" as an option and
allow people to write in other
ideas.
3. A double-barreled question is a
question that asks about more than
one issue, but only asks for one
response. An example of this
regarding brushing your teeth would
be: "How important is the time spent
brushing and number of brushes per
day?" Time spent brushing your
teeth is a different idea from the
number of times you brush your
teeth. If you want to know about
each of these ideas, it would be
better to ask them separately.

Record your
ideas
Write 2 survey questions that would
be given to teenagers with regard to
listening to music.

One question that has some bias

discussed in this section and one
question that is an effective question
with minimal bias.

Explain what type of bias is present

in the one question and why the
other question would be effective.

Response Bias
Definition: Response Bias is a type of
measurement bias where the survey
respondents were influenced to answer in
a certain way often because the survey is
not anonymous or because the
respondent thought that they had to
answer a certain way.

Example
A teacher may check that the class
understands something by asking the
students if they have any questions. If
no hands go up, the teacher assumes
the students understand. This
question may have response bias as
there may be students who did not
want to say they didn't understand in
front of the entire class.

Making a survey anonymous is a good

way to limit response bias, as
respondents will be comfortable stating
their true beliefs or opinions. Even with
making the survey anonymous, some
people may still tend to answer questions
the way they feel the surveyor wants
them to be answered. It is best if
questions are written in a way where
people feel that all answers are equally
accepted.

Designing a Survey
It is very important that a survey is written
to minimize measurement bias so that
the true beliefs of the sample
respondents are measured.

Types of Questions
In this activity you are exposed to a lot of
different types of questions to collect
data. For information on more specific
types of questions, read the following
article from QuestionPro.

If you want to view any links in this

pdf, right click and select "Open
Link in New Tab" to avoid leaving
this page. (View the original article.)

Writing Effective Questions

In addition to the knowledge of different
types of measurement bias, a survey
writer needs to also keep the following in
mind.

1. Ethics: When designing questions,

it is important that the survey writer is
in the correct frame of mind and is
looking to get the true beliefs of the
sample. If the goal of the survey is
to measure what people actually
think, rather than get people to say
what we'd like them to think, this will
help avoid phrasing questions in
ways that manipulate the opinion of
the respondents. This isn't only the
right thing to do, it's the best thing to
do in the long run.
Example
Say you are a marketing
consultant for Coca-Cola. You
want to know if people would like
a chocolate flavoured soda. You
ask questions that suggest that
people are overwhelmingly in
favour of this by being vague and
focusing on the appealing aspects
of chocolate and soda. If the
conclusion is that people will love
this, it may result in a lot of time
and money going into developing
the product, only to find out later
that people don't want it at all.

2. Writing Demographic Questions:

When asking personal questions, it
is important to ask if you really need
this information. It is very important
to do your own research as to how to
ask the particular question you want
to ask. Asking the question in the
wrong way may result in people not
answering your survey or continuing
it but not answering truthfully. It is
important to get a truthful answer
from all respondents.
Example
When asking for gender, it is best
practice to ask an open ended
question such as "To which
gender identity do you most
identify?"

3. Cultural Bias: It is important to ask

yourself whether or not some of your
questions may result in cultural bias.
Cultural bias is when respondents
tend to answer surveys in different
ways depending on their culture or
nationality. As an example, consider
the Likert scale. This is just one
example of cultural bias, and each
survey question needs to be
evaluated for whether or not the true
beliefs of the respondents are being
measured.
Example
The Likert scale was invented by
American psychologist Rensis
Likert. The typical Likert scale
asks respondents to agree with a
statement with a choice of the
following:

Strongly Disagree,
Disagree,
Neither Agree or Disagree,
Agree,
Strongly Agree.
Other rating scales can be used
that ask people to rate on a scale
from a min to max (from 1 to 10,
for example).

There may be a tendency for

some cultures to avoid saying
"Strongly" and for other cultures to
be more likely to answer
"Strongly."

4. Randomize the Questions: There

is chance that people may answer
questions in a different way
depending on what order they are
being asked the questions in. One
way to limit this bias is to randomize
the questions, if possible, so that
they appear in a different order for
each respondent. This will make the
error that would be caused by the
order of the questions more random.
This also requires us to use
technology to administer surveys.

Creating an Online
Survey
The following example is of a survey
created using Google Forms. There are
other internet tools that will allow you to
create a survey, with functionality that will
allow you to minimize the measurement
bias. Consider the Self-Assessment from
last activity.

Example
Take a moment to assess your
achievement of the success criteria
and your learning skills.

Questions

1. I can identify and describe the

different types of sampling
techniques.
a. Yes
b. Almost
c. Not Yet
2. I can identify and describe the
various types of bias that can
occur when sampling.
a. Yes
b. Almost
c. Not Yet
3. I can design an effective method
for getting an appropriate sample
from a population.
a. Yes
b. Almost
c. Not Yet
4. Evaluating my Responsibility: Did
I share my ideas on the criteria for
how to develop an experimental
and observational study with the
class?
a. Yes
b. Somewhat
c. No
5. Evaluating my Independent Work:
Did I accept others ideas and use
them to get insight of my own?
a. Yes
b. Somewhat
c. No

Watch the following video for a

demonstration on how to create the
survey, shorten the url so that it is easier
to type in, and look at responses in a
spreadsheet:

Creating A Survey In Googl…

Consider the
Following
Respond to one of the following
questions:

1. Do any of the five survey

questions from the self-
assessment of activity five
contain bias? Based on the
concepts of this activity,
describe the bias and a way to
improve the question so that it is
more effective.

2. Find a biased survey question

online. Based on the concepts of
this activity, describe the bias
and a way to improve the
question so that it is more
effective.

3. Create a biased survey

question. Based on the concepts
of this activity, describe the bias
and a way to improve the
question so that it is more
effective.