Lecture 5: Collecting data

1
Collect data
Who
What

Evaluation
questions

Sources of
Indicators: information:
Evidence that METHODS
program records,
answers your
individuals,
questions
public

2
Source of evaluation information

• Existing information
• People
• Pictorial records and observations

3
Quantitative: numbers breadth generalizability
Qualitative: words depth specific

Remember, "Not everything that counts can be counted."

4
Paradigm dimensions

Qualitative Quantitative
• Naturalistic inquiry • Scientific/experimental design
• Holistic, system-wide • Independent, dependent variables
perspective • Standardized, uniform
• Uniqueness and diversity • Deductive reasoning
• Inductive reasoning • Quantitative data (numbers)
• Qualitative data (words) • Quantitative methods − structured,
• Qualitative methods − standardized
unstructured, open-ended • Probabilistic, random sampling
• Purposeful sampling • Fixed, controlled design
• Emergent, flexible design • Statistical analysis
• Content analysis • Generalizations
• Extrapolations

5
Quantitative methods –
Qualitative methods
Quantitative Qualitative

Surveys Focus groups

Questionnaires
Tests Unstructured
interviews
Existing databases Unstructured
observations

6
Often, it is better to use more than one
method….
Mixed methods for one program
• Log of activities and participation
• Self-administered questionnaires
completed after each workshop
• In-depth interviews with key
informants
• Observation of workshops
• Survey of participants

7
Are the data reliable and valid?

• Validity: Are you measuring what you think

you are measuring?
– Example:
• Reliability: if something was measured
again using the same instrument, would it
produce the same (or nearly the same)
results?
– Example:

8
“Trustworthy” and “credible” data

What do these words mean relative to

your evaluation information?

How can you help ensure that your

evaluation data are trustworthy and
credible?

9
Common data collection methods

• Survey • Testimonials
• Case study • Tests
• Interview • Photographs,
• Observation videotapes, slides
• Group assessment • Diaries, journals,
logs
• Expert or peer
reviews • Document review
and analysis
• Portfolio reviews

10
When choosing methods, consider…

•The purpose of your evaluation − Will the

method allow you to gather information that
can be analyzed and presented in a way that
will be credible and useful to you and others?

•The respondents − What is the most

appropriate method, considering how the
respondents can best be reached, how they
might best respond, literacy, cultural
considerations, etc.?
11
Consider…
• Resources available. Time, money, and staff to
design, implement, and analyze the information.
What can you afford?
• Type of information you need. Numbers, percents,
comparisons, stories, examples, etc.
• Interruptions to program or participants. Which
method is likely to be least intrusive?
• Advantages and disadvantages of each method.
• The need for credible and authentic evidence.
• The value of using multiple methods.
• The importance of ensuring cultural
appropriateness.

12
Quality criteria for methods

UTILITY
Will the data sources and collection
methods serve the information needs
of your primary users?

13
Quality criteria…

FEASIBILITY
Are your sources and methods
practical and efficient?
Do you have the capacity, time, and
resources?
Are your methods non-intrusive and
non-disruptive?

14
Quality criteria…

PROPRIETY
Are your methods respectful, legal,
ethical, and appropriate?
Does your approach protect and
respect the welfare of all those
involved or affected?

15
Quality criteria…
ACCURACY
Are your methods technically adequate to:
• answer your questions?
• measure what you intend to measure?
• reveal credible and trustworthy
information?
• convey important information?

16
There is no one right method of
collecting data.

Each has a purpose, advantages, and

challenges.

The goal is to obtain trustworthy,

authentic, and credible evidence.

Often, a mix of methods is preferable.

17
Culturally appropriate evaluation methods
• How appropriate is the method given the
culture of the respondent/the setting?
• Culture differences: nationality, ethnicity,
religion, region, gender, age, abilities, class,
economic status, language, sexual
orientation, physical characteristics,
organizational affiliation

18
Is a written questionnaire culturally appropriate?

Things to consider:
• Literacy level
• Tradition of reading, writing
• Setting
• Not best choice for people with oral tradition
• Translation (more than just literal translation)
• How cultural traits affect response – response sets
• How to sequence the questions
• Pretest questionnaire may be viewed as intrusive

19
Are interviews culturally appropriate?
Things to consider:
• Preferred by people with
an oral culture
• Language level proficiency;
verbal skill proficiency
• Politeness – responding to authority (thinking it’s
unacceptable to say “no”), nodding, smiling,
agreeing
• Need to have someone present
• Relationship/position of interviewer
• May be seen as interrogation
• Direct questioning may be seen as impolite,
threatening, or confrontational

20
Are focus groups culturally appropriate?
Things to consider:
• Issues of gender, age, class, clan differences
• Issues of pride, privacy, self-sufficiency, and
traditions
• Relationship to facilitator as prerequisite to
rapport
• Same considerations as for interview

21
Is observation culturally appropriate?
Things to consider:
• Discomfort, threat of being observed
• Issue of being an “outsider”
• Observer effect
• Possibilities for
misinterpretations

22
Cultural issues related to use of existing
data/records
• Need careful translation of documents in
another language
• May have been written/compiled using
unknown standards or levels of aggregation
• May be difficult to get authorization to use
• Difficult to correct document errors if low
literacy level

23
Culturally appropriate informed consent

How can we be culturally sensitive and

respectful and ensure the protection of
those involved in our evaluations?
– Children
– Marginalized, “less powerful” participants

24
Focus groups

Structured small group interviews

“Focused” in two ways:
– Persons being interviewed are similar in
some way (e.g. limited resource
families, family services professionals,
or elected officials).
– Information on a particular topic is
guided by a set of focused questions.

25
Focus groups

Focus groups are used...

• To solicit perceptions, views, and a
range of opinions (not consensus)
• When you wish to probe an issue or
theme in depth

26
Survey

A structured way to collect information

using questionnaires. Surveys are
typically conducted through the mail
(electronic or surface), phone, or
internet.

27
Survey

Surveys are used…

• To collect standardized information
from large numbers of individuals
• When face-to-face meetings are
inadvisable
• When privacy is important or
independent opinions and responses
are needed

28
Steps in planning a survey

1. Decide who should be involved in the process.

2. Define survey content.
3. Identify your respondents.
4. Decide on the survey method.
5. Develop the questionnaire.
6. Pilot test the questionnaire and other materials.
7. Think about analysis.
8. Communicate about your survey and its results.
9. Develop a budget, timeline, and management
process.

29
Response rate

The proportion of people who respond:

divide the number of returned surveys
by the total number of surveys
distributed.
Example: If you distribute 50
questionnaires and you get 25
questionnaires back, your response
rate is 50%.

30
Response rate

# that answered
= response rate
# you contacted

31
Response rate

• High response rate promotes

confidence in results.

• Lower response rate increases the

likelihood of biased results.

32
Response rate
•There is no standard response rate. “The
higher, the better.” Anything under 60% is
a warning.
•Why is high return important? It’s the only
way to know if results are representative.
•Address low response. How are people
who didn’t respond different from those
who did? Only describe your results in
terms of who did respond.

33
How to increase response rate
• Generate positive publicity for your survey.
• Over sample.
• Ensure that respondents see the value of
participating.
• Use a combination of methods.
• Make (multiple) follow-up contacts.
• Provide incentives.
• Provide 1st class postage/return postage.
• Set return deadlines.
• Make the survey easy to complete.
34
If response rate is low…

Use language that is suggestive rather

than decisive.
Examples: “The data suggests” vs.
“These data show”; “It appears” vs.
“We can conclude”
• Don’t generalize findings to the entire
group.
• Clearly describe who the data
represents.
35
Document review
Using information that already exists in
records, receipts, meeting minutes, reports,
budgets…rather than collecting new data

There is a wealth of information available on

the web.

CHECK − What information is already

available?

36
Document review − Sources
National Center for Education Statistics
http://nces.ed.gov
Census Bureau http://www.census.gov
Bureau of Labor Statistics http://stats.bls.gov
National Center for Health Statistics
http://cdc.gov/nchs/
Children’s Defense Fund
http://www.childrensdefense.org
Wisconsin Department of Public Instruction
Local school districts
ERIC searches http://www.eric.ed.gov/
County government

37
Document review –
Advantages of using existing data

• Available – don’t have to collect data

• Low cost
• Minimum staff required
• Comparative or longitudinal data may
be available

38
Document review −
Issues in using existing data

• Missing or incomplete data

• Confidentiality issues
• Unknown, different, or changing
definitions of data make comparison
difficult
• May not match what you need in terms
of geographic location, same time
period, or population – may be too
aggregated
39
Observation…

• Is watching people, programs,

events, communities, etc.
• Involves all 5 senses: sight, hearing,
smell, touch, and taste
– observation includes more than just
“seeing”

40
Observation is used…

• To provide information about real-life

situations and circumstances
• To assess what is happening
• Because you cannot rely on
participants’ willingness and ability to
furnish information

41
When is observation useful?

• When you want direct information

• When you are trying to understand
an ongoing behavior, process,
unfolding situation, or event
• When there is physical evidence,
products, or outcomes that can be
readily seen
• When written or other data collection
methods seem inappropriate

42
Observations
• Advantages • Disadvantages
– Most direct measure – May require training
of behavior – Observer’s presence
– Provides direct may create artificial
information situation
– Easy to complete, – Potential for bias
saves time – Potential to overlook
– Can be used in meaningful aspects
natural or – Potential for
experimental settings misinterpretation
– Difficult to analyze

43
Observation – Purpose, benefits
• Unobtrusive
• Can see things in their natural context
• Can see things that may escape conscious
awareness, things that are not seen by others
• Can discover things no else has ever really paid
attention to, things that are taken for granted
• Can learn about things people may be unwilling to talk
about
• Inconspicuous – least potential for generating
observer effects
• Least intrusive of all methods
• Can be totally creative – has flexibility to yield insight
into new realities or new ways of looking at old
realities

44
Observation – Limitations
1. Potential for bias
• Effect of culture on what you observe and
interpret
2. Reliability
• Ease of categorization

Usually you do not rely on observation

alone; combine your observations with
another method to provide a more
thorough account of your program.

45
Observation – Ethical issues
• Unobtrusiveness is its greatest
strength; also potential for abuse in
invasion of privacy
• Can venture into places and gather
data almost anywhere
• Covert – overt
– Always consider ethics and human
subjects protection.

46
Types of observation

Structured Unstructured
Looking for Looking at

Observing what does not happen may be as

important as observing what does happen.

47
Steps in planning for observation
• Determine who/what will be observed.
• Determine aspects that will be observed
(characteristics, attributes, behaviors, etc.).
• Determine where and when observations will be
made.
• Develop the observation record sheet.
• Pilot test the observation record sheet.
• Train the observers and have them practice.
• Collect the information.
• Analyze and interpret the collected information.
• Write up and use your findings.

48
Who/what to observe
• People (individuals, groups,
communities)
– Characteristics
– Interactions
– Behaviors
– Reactions
• Physical settings
• Environmental features
• Products/physical artifacts

49
Observation – Example

If you want
information about… You would record…
Who uses a particular Total number of users
service broken down by gender,
age, ethnicity, etc.
Interactions between # and types of questions
youth and adults asked by each
Neighborhood safety ???

50
What to observe − Example
Exhibit on tobacco use at a county fair

Information needed:
Number of youth who visit the exhibit: age, gender,
cultural background

Can the information be observed accurately?

E.g., gender may be more obvious than age
or cultural background.
Will the observer affect the situation?

51
Example – Plans for observing
participation in an after school program
• Who: youth attending the program
• What:
– approximate age
– gender, cultural background
– length of time student stays in the program
• When: all hours the program is open for
one week each month during 2007

52
Recording your observations
Observations need to be recorded to
be credible. You might use:
– Observation guide
– Recording sheet
– Checklist
– Field note
– Picture
– Combination of the above

53
Observational rating scales

• Written descriptions – written

explanations of each gradation to
observe
• Photographs – series of photos that
demonstrate each of the grades on
the rating scale
• Drawings, sketches, etc. – other
visual representations of conditions
to be observed
54
Who are the observers?
• You – program staff
• Participants
• Stakeholders
• Colleagues
• Volunteers
• College students

55
Training observers

Training is often necessary:

– To learn what to look for
– To learn how to record observations
– To practice
– When want standardized observations
across sites: important that all
observers use same methods, rate
same observation in same way

56
Practice

For our workshop today, what

observational data could we collect
that would tell us …
– whether learning is occurring
– the characteristics of attendees
– whether the setting is conducive to
learning
– whether the materials are easy to use

57
Practice

Imagine you are sitting in a room

where ten youth are participating in a
computer demonstration. If you were
looking for indicators of student
interest and learning from the
demonstration, what would you look
for?
(Remember to include verbal and nonverbal
indicators.)

58
Interviewing is…

• Talking and listening to people

• Verbally asking program participants the
program evaluation questions and hearing
the participant’s point of view in his or her
own words. Interviews can be either
structured or unstructured, in person or
over the telephone.
• Done face-to-face or over the phone
• Individual; group

59
Interviews are useful…

• When the subject is sensitive

• When people are likely to be inhibited
in speaking about the topic in front of
others
• When people have a low reading ability
• When bringing a group of people
together is difficult (e.g., in rural areas)

60
Interviews

Verbally asking program participants

the program evaluation questions and
hearing the participant’s point of view
in his or her own words.
Interviews can be either structured or
unstructured, in person or over the
telephone.

61
Interviews

• Advantages • Disadvantages
– deep and free – costly in time and
response personnel
– flexible, – requires skill
adaptable – may be difficult to
– glimpse into summarize
respondent’s responses
tone, gestures – possible biases:
– ability to probe, interviewer,
follow-up respondent, situation

62
Types of interviewing

Structured Conversational

63
Type: Structured interview

• Uses script and questionnaire

• No flexibility in wording or order of
questions
• Closed response option
• Open response option

64
Type: Guided interview

• Outline of topics or issues to cover

• May vary wording or order of
questions
• Fairly conversational and informal

65
Type: Conversational interview

• May not know that an interview is

taking place
• Spontaneous
• Questions emerge from the situation
and what is said
• Topics or questions are not
predetermined
• Individualized and relevant to situation
66
Probing
Interview question:
“What did you like best about this program?”
Response: “I liked everything.”
Probe 1: “What one thing stood out?”
R: “Being with my friends.”
Probe 2: “What about the program activities?”
R: “I liked it when we worked as a team.”
Probe 3: “How come?”
R: “It was neat to hear each other’s perspectives. I
heard some things I hadn’t considered before.”
Probe 4: “What is one thing that you learned?”

67
Interviewing tips
• Keep language pitched to that of
respondent
• Avoid long questions
• Create comfort
• Establish time frame for interview
• Avoid leading questions
• Sequence topics
• Be respectful
• Listen carefully

68
Recording responses

• Write down response

• Tape record
• Key in on computer
• Work in pairs
• Complete notes after interview

69
Questionnaires are…
• Data collection instruments used to collect
standardized information that can be
expressed numerically or through short
answers
• Basic instruments of surveys and structured
interviews
• Appropriate when…
– you want information from many people
– you have some understanding of the situation
and can ask meaningful questions
– information is sensitive or private − anonymous
questionnaires may reduce bias
70
Questionnaires

• Advantages • Disadvantages
– can reach large – might not get
numbers careful feedback
– provide for – wording can bias
anonymity client’s response
– relatively – response rate is
inexpensive often low
– easy to analyze – literacy demands

71
When should a questionnaire be used?
• Respondents can provide useful
information about the topic.
• You know what it is you want to know and
are reasonably sure that you can ask
standardized questions to get the
information.
• Respondents can be relied upon to provide
the information you need (perhaps with
incentives). This means they can
comprehend the questions and respond
properly, they are truthful, and they are
motivated enough to respond carefully.

72
Good questionnaires are NOT EASY!

• Developing a good questionnaire,

takes time, time, and more time.
• Multiple (even a dozen!) drafts may
be involved before the questionnaire
is ready.
• It’s important to involve others in
writing the questionnaire.

73
Questionnaire design − Considerations
• Kind of information: What do you want to
know? Is the information already
available?
• Wording of questions and responses
• Formatting the questionnaire
• Pre-testing
• Cover letters and introductions
• When/where will the questionnaire be
distributed?
• How will returns be managed? How will
the data be analyzed?
• Who is responsible for each task?
74
Questionnaire design

• Is the information already available?

• Don’t ask a question unless
it has a use.
– Eliminate the “nice to know.”
• What will you do with each piece of
information gathered?

75
Questionnaire design
• Write questions through your
respondent’s eyes.
– Will the question be seen as
reasonable?
– Will it infringe on the respondent’s
privacy?
– Will the respondent be able and willing
to answer the question?
• Be selective and realistic when
writing questions.
76
6 STEPS IN DEVELOPING EFFECTIVE
QUESTIONNAIRES
1. Decide what information you need.
2. Determine sample – respondents.
3. Develop accurate, user-friendly
questionnaire.
4. Develop plan for distribution, return,
and follow-up.
5. Provide clear instructions and a
good cover letter.
6. Pilot test.
77
Step 1: What information is needed?

• Be specific
• Need to know vs. would like to know
• Check to see if information exists
elsewhere
• What do you want to be able to say:
counts, percentages, relationships,
narratives

78
Step 2: Sample

• Who will complete the

questionnaire?
• What do you know about their
preferences, abilities, and cultural
characteristics that may affect the
way they respond?

79
Step 3: Develop questionnaire

• Make sure questions cover

information needed.
• Word questions carefully.
• Consider cultural nuances.
• Sequence questions appropriately.
• Attend to formatting.

80
Step 3 continued

• Write clear, complete directions.

• Review to see if it is user-friendly;
consider the respondent.
• Make the questionnaire attractive.
• Work as a team.
• Plan on writing several draft
questionnaires.

81
Step 4: Plan distribution, return, follow-up

Distribution: when, where

– At meetings, sites, through mail, email,
internet
Return: when, where
– Return to individual, collection box
– Return envelope addressed/stamped
– Return envelope addressed only
Follow-up

82
Step 5: Cover Letter − Explanation
• Purpose of questionnaire –
how information will be used
• Why they are being asked to fill it out
• Importance of their response
• How and when to respond
• Whether response will be anonymous or
confidential
• Your appreciation
• Promise results, if appropriate
• Signature − sponsorship

83
Step 6: Pilot test
• Always
• With people as similar to respondents
as possible
– Do they understand the questions? The
instructions?
– Do questions mean same thing to all?
– Do questions elicit the information you
want?
– How long does it take?
• Revise as necessary
84
Kinds of information –
What do you want to know?
• Knowledge − what people know,
how well they understand something
• Beliefs − attitudes, opinions
• Behaviors − what people do
• Attributes/Demographics − what
people are and what people have

85
 Change in knowledge
Impact of divorce on children
As a result of this program, to what extent do you
understand the following about children and divorce:

Not well Somewhat Very well Already knew

a. Stages of 1 2 3 4
grief

b. Self-blame 1 2 3 4
or guilt

c. The desire 1 2 3 4
for parents to
reunite

86
Change in skills

Communication skills
List three communications techniques
you learned in this course that you
have used with your children:
1.
2.
3.

87
Change in attitude

As a result of this course, to what extent

do you feel that your attitude has
changed about:
a. Discussing your children with your ex
not at all somewhat a great deal
b. Allowing your former in-laws to see your
children
not at all somewhat a great deal

88
Change in behavior
How visitation disputes are handled

1. Describe how you and your ex-spouse

handled visitation disagreements
before the course.

2. Describe how you and your ex-spouse

have handled visitation disagreements
since the workshop.

89
Attributes –
What people are, what people have

Demographic characteristics − age,

education, occupation, or income
– Where do you currently live?
– How many children do you have?
– What is your age?
– How many years have you been
employed at your current job?

90
Types of questions

• Open-ended questions − allow

respondents to provide their own
answers

• Closed-ended questions − list

answers and respondents select
either one or multiple responses

91
Open-ended questions

• Do not provide any specific

responses from which the participant
would choose.

• Allow respondents to express their

own ideas and opinions.

92
Open-ended questions

Pros: Cons:
• Can get • More difficult to
unintended or answer
unanticipated • May be harder to
results categorize for
• Wide variety of interpretation
answers • More difficult for
• Answers in people who don’t
participants’ write much
“voices”
93
Open-ended questions

Examples:

What communication skills did you

learn in this workshop that you will
use with your children?

What benefits do you receive from

this organization?

94
Closed-ended questions

• Provide specific answers from which

the participant must choose.
• Sometimes called “forced choice.”
• Response possibilities include: one
best answer, multiple responses,
rating, or ranking scale.

95
Closed-ended questions

Pros: Cons:
• Easy to analyze • Chance of none of
responses the choices being
• Stimulates recall appropriate
• Biases response to
what you’re looking
for
• Misses unintended
outcomes

96
Closed-ended questions
Example − one best answer:

What does the word “nutrition” mean to you?

(Circle one number.)

1 Getting enough vitamins

2 The food you eat and how your body uses it
3 Having to eat foods I don’t like
4 Having good health

97
Closed-ended questions
Example − multiple responses:
Of the communication skills taught in this
workshop, which will you use with your
children? (Check all that apply.)
active listening
acknowledge feelings
ask more open-ended questions
provide one-on-one time for discussion
negotiation
other

98
Closed-ended questions
Example − rating scale
To what extent do you agree or
disagree with the new zoning code?
(Circle one.)
1 Strongly disagree
2 Mildly disagree
3 Neither agree or disagree
4 Mildly agree
5 Strongly agree
99
When wording the questions, consider…

• The particular people for whom the

questionnaire is being designed
• The particular purpose of the
questionnaire
• How questions will be placed in
relation to each other in the
questionnaire

100
Use clear, specific, simple wording.

• Match vocabulary and reading

skills of your respondents.
• Are any words confusing? Do any
words have a double meaning?
• Avoid the use of abbreviations and
jargon.

101
 Example:
Use clear, specific, simple wording.

• Avoid jargon or technical language.

Jargon:
What kind of post-litigation concerns have
you and your ex-spouse had?
Better:
Since having your visitation rights set by a
judge, what other concerns have you and
your ex-spouse had about visitation?

102
Include all necessary information.

• Avoid vague questions and answers.

• Avoid ambiguous words or phrases.
• Avoid questions that may be too
specific.
• Avoid making assumptions.

103
Example: Vague questions

Vague:
How will this seminar help you?

104
Example: Vague questions
Better:
What skills did you learn in this
seminar that will help you follow the
child custody arrangements set by
the court?
how to negotiate changes or with my ex-spouse
how to explain visitation arrangements to my
children
steps to requesting a change in arrangements
from the court
how to separate child support from visitation
disputes

105
Example: Avoid ambiguous words or
phrases.
Ambiguous:
How has your child demonstrated
improved communication skills
since participating in “Let’s
Communicate”?

106
Example: Avoid specificity that limits the
potential for reliable responses.

Too specific:
How many meals have you eaten as a
family during the past year?
number of meals

107
Example: Avoid making assumptions.

Question for teachers that makes

assumptions:

What practices have you used to

get more parents to read to their
children?

108
Avoid leading questions.

• Biased questions
– Influence people to respond in a certain
way
– Make assumptions about the respondent
– Use language that has strong positive or
negative appeal

109
Example: Leading questions
Leading:
Do you think this seminar will help
you stop fighting with your spouse
about the children?
Better:
How do you think this seminar will
help you work with your spouse to
address your children’s concerns?

110
Avoid double-barreled questions.

• Ask one question at a time.

• Avoid ambiguity − questions that have
multiple responses.

111
Example: Double-barreled question

Double:
How will this seminar help you communicate
better with your children and their
grandparents about your divorce?

Better:
How will this seminar help you communicate
with your children about your divorce?
How will this seminar help you communicate
with your children’s grandparents about their
relationship with their grandchildren?

112
Make the response categories clear,
logical, and mutually exclusive.

• Only one possible answer

• Similar-sized categories
• Responses in a logical order

113
Example: Clear, logical, and mutually
exclusive responses
Poor spacing and Better spacing, logic, and
logic: mutually exclusive:
Children’s Ages Children’s Ages
0−1 under 1 year of age
1−3 1−3 years of age
3−6 4−6 years of age
7−12 7−9 years of age
13−18 10−12 years of age
13−15 years of age
16−18 years of age

114
Example: Vague quantifier
Vague:
How often did you attend an
Extension-sponsored workshop
during the past year?
a. Never
b. Rarely
c. Several times
d. Many times
115
Example: Vague quantifier

Better:
How often did you attend an
Extension-sponsored workshop
during the past year?
a. Not at all
b. One to two times
c. Three to five times
d. More than five times
116
Rating scales
• Ordered options to gauge difference of
opinion.
• Keep the order of choices the same
throughout the form.
• Odd number of options allows people
to select a middle option.
• Even number forces respondents to
take sides.
• Simpler is better.
117
Types of rating scales

Category scales
Numeric scales
Semantic differentials

118
Category/Rating scales

• Use words or phrases to express a

range of choices.
• The number of categories depends
on the amount of differentiation.
• Three, four, or five categories are
most common.

119
Category/Rating scales
• Balance the scale with an equal
number of positive and negative
options.
• “No opinion” or “uncertain” are not
part of a scale. They are usually
placed off to the side or in a separate
column.
• All choices should refer to the same
thing/concept.
120
Category/Rating scales − Example
Poor: Better:
Not worth my time Not at all
Slightly interested interested
Moderately Slightly interested
interested Moderately
Very interested interested
Very interested

Left column includes two concepts –

“worth” and “interest level.”

121
Rating scales − Words

Not much A little Not much

Some Some Little
A great deal A lot Somewhat
Much
A great deal

122
Rating scales − Words

Never Extremely poor

Seldom Below average
Often
Average
Always
Above average
Excellent

123
Rating scales − Words

Strongly Disagree Completely

disagree Neither agree disagree
Disagree nor Mostly disagree
Agree disagree Slightly
Strongly Agree disagree
agree Slightly agree
Uncertain Mostly agree
Completely
agree

124
Formatting considerations

• Overall appearance
• Length of the questionnaire
• Order of questions
• Demographic data collection

125
Formatting − Overall appearance

• Use an easy-to-read typeface.

• Leave plenty of white space.
• Separate different components of a
questionnaire by using different type
styles.
• Use arrows to show respondents
where to go.

126
Formatting − Length of the questionnaire

• Shorter questionnaires usually generate

higher return.
• Include enough items to be thorough
but don’t over-burden the respondent.
• Length is not usually as important as
other formatting characteristics.

127
Formatting − Order of questions
• Introduction − Include questionnaire’s
sponsor, purpose, use, confidentiality, etc.
• Include instructions for how to answer the
questions (e.g., Circle one; Check all that
apply).
• Arrange questions so they flow naturally.
• Place demographic questions at the end of
the questionnaire.
• Be consistent with numbers, format, and
scales.

128
Formatting − Order of questions

• Start with the easiest questions −

avoid controversial topics.
• Address important topics early.
• Move from specific questions to
general questions.
• Move from closed-ended to
open-ended questions.

129
Formatting − Demographic data collection

• Only include questions about

demographic data that you will use.
• You may want to preface
demographic questions with the
purpose for collecting the information.
• You may need to state that providing
this information is optional and/or
explain how it affects program
eligibility.
130
Formatting − Demographic data collection
Age Residence
Gender Previous contact with
Ethnicity organization
Marital status Prior knowledge of
Family size topic
Occupation First-time participant
vs. repeats
Education
How you learned
Employment status
about the program

131
Pre-test the questionnaire

ALWAYS
ALWAYS
ALWAYS

132
What you want to find out in a pretest:
• Does each question measure what it is
supposed to measure?
• Are all the words understood?
• Are questions interpreted in the same
way by all respondents?
• Are all response options appropriate?
• Is there an answer that applies to each
respondent?
-Salant and Dillman (1994)
Source: Salant, P., & Dillman, D. A. (1994). How to conduct your own survey. New York: John Wiley & Sons, Inc.

133
Pre-testing questions
• Are the answers respondents can choose
from correct? Are some responses
missing?
• Does the questionnaire create a positive
impression – does it motivate people to
answer it?
• Does any aspect of the questionnaire
suggest bias?
• Do respondents follow the directions?
• Is the cover letter clear?
-Salant and Dillman (1994)
Source: Salant, P., & Dillman, D. A. (1994). How to conduct your own survey. New York: John Wiley & Sons, Inc.

134
Pre-testing steps
1. Select reviewers who are similar to
the respondents and who will be
critical. (Also ask your colleagues to
review it.)
2. Ask them to complete the
questionnaire as if it were “for real.”
3. Obtain feedback on the form and
content of the questionnaire and the
cover letter. Was anything confusing,
difficult to answer, de-motivating?

135
Pre-testing steps, continued

4. Assess whether the questions

produce the information you need.
5. Try the tabulation and analysis
procedures.
6. Revise.
7. If necessary, repeat these steps to
pre-test the revised version.

136
Revise and revise…

• A quality questionnaire is almost

never written in one sitting.
• A quality questionnaire goes through
multiple revisions (maybe a dozen!)
before it is ready.
• Remember – a list of questions is
just the starting point. There are
many factors that affect response.

137
Choices: Timing of data collection

When will data be collected?

– Before and after the program
– At one time
– At various times during the course of the
program
– Continuously through the program
– Over time − longitudinally

138
A good cover letter will include
information about…
• Purpose and importance of the
survey
• Survey sponsor − use letterhead
• Why the respondent was selected to
participate
• Benefit(s) of completing survey
• Assurance of anonymity or
confidentiality
139
A good cover letter will include
information about…

• How results will be used

• Instruction for returning the survey
• When to respond
• How to obtain survey results
• Contact information

140
Cover letters − Tips

• Personalize the letter in salutation or

signature
• Hand-sign the letter
• Express appreciation for their
participation
• Include pre-addressed, stamped
return envelope

141
Cover letters – Pre-test

• Remember to pre-test your cover

letter just as you pre-test your
questionnaire!

142
Sampling – The basics

• Why sample?
• What are options for sampling
design?
• What determines sample size?
• What should you consider when
conducting the sample?

143
Why a sample?

• Save money.
• Save time.
• Minimize error and maximize
representation.

144
Types of sampling strategies:
Probability: Nonprobability:
• Why? • Why?
Generalize to Generalizability not
as important. Want
population.
to focus on “right
Some examples: cases.”
– Simple random Some examples:
sample – Quota sample
– Stratified sample – “Purposeful” sample
– Cluster sample – “Convenience” or
– Systematic sample “opportunity”
sample
145
Things to remember
• The smaller the sample size, the greater the
variability.
• Goal of sampling: to reduce variability enough to say
it represents the population without increasing costs.
• Sampling bias: Sampling is not “really” random, but
ion a good sample:
1. Each unit should have an equal chance of
being chosen.
2. Choosing one unit should not affect whether
another is chosen.
• Response bias: Respondents with particular
characteristics tend to respond in particular ways.

146