UNIT VII

QUESTIONNAIRE DESIGN

7.1. THE FUNCTION OF A QUESTIONNAIRE

Questionnaires are quite flexible in what they can measure, however they are not equally suited
to measuring all types of data. We can classify data in two ways, Subjective vs. Objective and
Quantitative vs. Qualitative.

 When a questionnaire is administered, the researchers control over the environment will
be somewhat limited. This is why questionnaires are inexpensive to administer.
 This loss of control means the validity of the results is more reliant on the honesty of the
respondent. Consequently, it is more difficult to claim complete objectivity with
questionnaire data than with results of a tightly controlled lab test. For example, if a
group of participants are asked on a questionnaire how long it took them to learn a
particular function on a piece of software, it is likely that they will be biased towards
themselves and answer, on average, with a lower than actual time. A more objective
usability test of the same function with a similar group of participants may return a
significantly higher learning time.

 More elaborate questionnaire design or administration may provide slightly better

objective data, but the cost of such a questionnaire can be much higher and offset their
economic advantage. In general, questionnaires are better suited to gathering reliable
subjective measures,

 By their very nature, quantitative questions are more exact than qualitative. For example,
the word "easy" and "difficult" can mean radically different things to different people.

 Any question must be carefully crafted, but in particular questions that assess a
qualitative measure must be phrased to avoid ambiguity.

 Qualitative questions may also require more thought on the part of the participant and
may cause them to become bored with the questionnaire sooner. In general, we can say
that questionnaires can measure both qualitative and quantitative data well, but that
qualitative questions require more care in design, administration, and interpretation.

There is no all encompassing rule for when to use a questionnaire. The choice will be made
based on a variety of factors including the type of information to be gathered and the available
resources for the experiment. A questionnaire should be considered in the following
circumstances.

a. When resources and money are limited. A Questionnaire can be quite inexpensive to
administer. Although preparation may be costly, any data collection scheme will have
similar preparation expenses. The administration cost per person of a questionnaire can
 be as low as postage and a few photocopies. Time is also an important resource that
questionnaires can maximize. If a questionnaire is self-administering, such as an e-mail
questionnaire, potentially several thousand people could respond in a few days.
b. When it is necessary to protect the privacy of the participants. Questionnaires are
easy to administer confidentially. Often confidentiality is necessary to ensure participants
will respond honestly if at all. Examples of such cases would include studies that need to
ask embarrassing questions about private or personal behavior.
c. When corroborating (confirming) other findings. In studies that have resources to
pursue other data collection strategies, questionnaires can be a useful confirmation tools.
More costly schemes may turn up interesting trends, but occasionally there will not be
resources to run these other tests on large enough participant groups to make the results
statistically significant. A follow-up large scale questionnaire may be necessary to
corroborate these earlier results.

Questionnaires are versatile, allowing the collection of both subjective and objective data
through the use of open or closed format questions.. However, a questionnaire is only as good as
the questions it contains. There are many guidelines that must be met before your questionnaire
can be considered a sound research tool.

The majority deal with:

Making the questionnaire understandable and free of bias.

Mindful review and testing necessary to weed out minor mistakes that can cause great
changes in meaning and interpretation.

Must have a logical flow

Avoid too broad and generalized questions.

Eg. An objective such as "to identify points of user dissatisfaction with the interface and how
these negatively affect the software's performance" may sound clear and to the point, but it is
not. The questionnaire designer must clarify what is meant by user dissatisfaction. Is this
dissatisfaction with the learning of the software, the power of the software, of the ease of
learning the software? Is it important for the users to learn the software quickly if they learn it
well? What is meant by the software's performance? How accurate must the measurements be?
All of these issues must be narrowed and focused before a single question is formulated.

7.2. The questionnaire development process

Questionnaire design is a long process that demands careful attention.. Design begins with an
understanding of the capabilities of a questionnaire and how they can help your research. If it is
determined that a questionnaire is to be used, the greatest care goes into the planning of the
objectives.
Designing a good questionnaire always takes several steps such as: Defining the Objectives of
the survey, outlining the content of the questionnaire, selecting the question response format,
finally writing the Questionnaire.

 Determine what kind of questions do we ask? / Form/

 Determine the Question sequence

 Question formulation and wording

 Piloting your questionnaire

 Administering the questionnaire

7.2.1. What kind of questions do we ask? / Form/

Open-ended Questions

In general, there are two types of questions one will ask, open format or closed format. Open
format questions are those that ask for unprompted opinions. In other words, there are no
predetermined set of responses, and the participant is free to answer however he chooses.

Open format questions are good for soliciting subjective data or when the range of responses is
not tightly defined. An obvious advantage is that the variety of responses should be wider and
more truly reflect the opinions of the respondents. This increases the likelihood of receiving
unexpected and insightful suggestions, for it is impossible to predict the full range of opinion. It
is common for a questionnaire to end with an open format question asking the respondent for her
unabashed ideas for changes or improvements.

Example: Why did you enroll for this course?

………………………………………………………………………………

………………………………………………………………………………

Advantages of Open-Ended Questions:

 Greater freedom of expression

 No bias due to limited response ranges
 Respondent can qualify their answers

Disadvantages of Open-ended questions:

 First, their very nature requires them to be read individually. There is no way to
automatically tabulate or perform statistical analysis on them.
 More costly in both time and money
  They are also open to the influence of the reader, for no two people will interpret an
answer in precisely the same way.

 Finally, open format questions require more thought and time on the part of the
respondent.

 Whenever more is asked of the respondent, the chance of tiring or boring the respondent
increases.

 Researcher / interviewer may misinterpret (and therefore misclassify) a response

 Time consuming to code

Closed - ended Questions

Closed - ended questions usually take the form of a multiple-choice question.

 They are easy for the respondent; given there is no clear consensus on the number of
options that should be given in a closed format question.
 Obviously, there needs to be sufficient choices to fully cover the range of answers
but not so many that the distinction between them becomes blurred.

 Usually this translates into five to ten possible answers per questions. For questions
that measure a single variable or opinion, such as ease of use or liability, over a
complete range (easy to difficult, like to dislike), conventional wisdom says that
there should be an odd number of alternatives.

Types of Closed-ended questions

1. Name: Dichotomous

Description: Question offering two choices

Example: Did you watch television at all yesterday?

Yes / No

2. Name: Multiple choices

Description: Question offering three or more choices

Example: Which of these shops do you prefer?

Next / River Island / Gap Top Shop/ Top Man

3. Name: Likert scale

Description: Statement with which respondent shows the amount of agreement / disagreement

Example: Assessment by course-work is easier than assessment by examination

Neither agree nor

Strongly agree Agree Disagree  Strongly disagree
disagree

4. Name: Semantic differential scale

Description: Scale is inscribed between two bipolar words and respondent selects the point that
most represents the direction and intensity of his / her feelings

Example: The degree I am taking is.............

Interesting :_____:_____:_____:_____:_____:_____:_____: Boring

Useful :_____:_____:_____:_____:_____:_____:_____: Useless

Easy :_____:_____:_____:_____:_____:_____:_____: Difficult

5. Name: Rank order

Description: Respondent is asked to rate or rank each option that applies. This allows the
researcher to obtain information on relative preferences, importance etc. Long lists should be
avoided (respondents generally find it difficult to rank more than 5 items)

Example: Please indicate, in rank order, your preferred Chewing gum, putting 1 next to your
favorite through to 5 for your least favorite.

 Poppotine
 Strawberry
 Special mint
 Wow
 Banana

6. Name: Numeric scale

Description: Respondent specifies a particular value (can include decimal places)

Example: How far (to the nearest kilometre) did you travel today to reach this supermarket?
 ________km

Advantages of Closed-ended questions

 Saves time and money by restricting the answer set, it is easy to calculate percentages and
other hard statistical data over the whole group or over any subgroup of participants.
 Closed format questions also make it easier to track opinion over time by administering
the same questionnaire to different but similar participant groups at regular intervals.

 Finally closed format questions allow the researcher to filter out useless or extreme
answers that might occur in an open format question. Easy to code.

Disadvantages of Closed-Ended Questions

 Can draw misleading conclusions because of limited range of options

 Researcher / interviewer cannot deal with qualifications to responses e.g. "Yes, but….."
or "It depends" where only Yes/No are given as options

7.2.2. Determine the Question sequence

In order to make the questionnaire effective and ensure the quality to replies, researcher should
pay attention to the question sequence in preparing a questionnaire because it considerably
reduces the problem of understanding.

The following type of questions should generally be avoided as opening questions;

 Question that lead to great strain on the memory or intellect of the respondent
 Question of personal and intimate character

Also, as a general rule, question sequence should go from general to specific because the answer
to the given question is a function of not only the question itself but also to all previous questions
as well.

7.2.3. Question formulation and wording

All questions should have the following characteristics

 Easy to be understood
 Simple and not ambiguous

 Make it Specific

 The size should be kept to the minimum

 Concrete (conform) as much as possible to the respondents way of thinking

  Technical terms and vague expressions should be avoided

One example showing clarity of question to reduce interpretation problem

Clarity: This is probably the area that causes the greatest source of mistakes in
questionnaires. Questions must be clear, succinct, and unambiguous. The goal is to eliminate
the chance that the question will mean different things to different people. If the designer
fails to do this, then essentially participants will be answering different questions.

To this end, it is best to phrase your questions empirically if possible and to avoid the use of
necessary adjectives. For example, in asking a question about frequency, rather than
supplying choices that are open to interpretation such as:

 Very Often
 Often

 Sometimes

 Rarely

 Never

It is better to quantify the choices, such as:

 Every Day or More

 2-6 Times a Week

 About Once a Week

 About Once a Month

 Never

7.2.4. Piloting your questionnaire

Before you deliver any questionnaire, you should "pilot" it (i.e. test it) to check that it is going to
function effectively. There are a number of reasons why it is important to pilot a questionnaire:

 To test how long it takes to complete

 To check that the questions are not ambiguous

 To check that the instructions are clear

 To allow you to eliminate questions that do not yield usable data

7.2.5. Administering the questionnaire

Finally you make use of your questionnaire in order to collect a valid and reliable data that will
help you answer your research question.

7.2.6. Qualities of a Good Question

There are good and bad questions. The qualities of a good question are as follows:

1. Evokes the truth. Questions must be non-threatening. When a respondent is concerned about
the consequences of answering a question in a particular manner, there is a good possibility that
the answer will not be truthful. Anonymous questionnaires that contain no identifying
information are more likely to produce honest responses than those identifying the respondent. If
your questionnaire does contain sensitive items, be sure to clearly state your policy on
confidentiality.

2. Asks for an answer on only one dimension. The purpose of a survey is to find out
information. A question that asks for a response on more than one dimension will not provide the
information you are seeking. For example, a researcher investigating a new food snack asks "Do
you like the texture and flavor of the snack?" If a respondent answers "no", then the researcher
will not know if the respondent dislikes the texture or the flavor, or both. Another questionnaire
asks, "Were you satisfied with the quality of our food and service?" Again, if the respondent
answers "no", there is no way to know whether the quality of the food, service, or both were
unsatisfactory. A good question asks for only one "bit" of information.

3. Can accommodate all possible answers. Multiple choice items are the most popular type of
survey questions because they are generally the easiest for a respondent to answer and the easiest
to analyze. Asking a question that does not accommodate all possible responses can confuse and
frustrate the respondent. For example, consider the question:

What brand of computer do you own?

       A.IBM,PC
       B. Apple

Clearly, there are many problems with this question. What if the respondent doesn't own a
microcomputer? What if he owns a different brand of computer? What if he owns both an IBM
PC and an Apple? There are two ways to correct this kind of problem.

The first way is to make each response a separate dichotomous item on the questionnaire. For
example:

Do you own an IBM PC? (circle: Yes or No)

Do you own an Apple computer? (circle: Yes or No)

Another way to correct the problem is to add the necessary response categories and allow
multiple responses. This is the preferable method because it provides more information than the
previous method.

What brand of computer do you own?

(Check all that apply)

__ Do not own a computer

__IBM,PC
__Apple
__ Other

4. Has mutually exclusive options. A good question leaves no ambiguity in the mind of the
respondent. There should be only one correct or appropriate choice for the respondent to make.
An obvious example is:

Where did you grow up? __

A.country
B.farm
C. city

A person who grew up on a farm in the country would not know whether to select choice A or B.
This question would not provide meaningful information. Worse than that, it could frustrate the
respondent and the questionnaire might find its way to the trash.

5. Produces variability of responses. When a question produces no variability in responses, we

are left with considerable uncertainty about why we asked the question and what we learned
from the information. If a question does not produce variability in responses, it will not be
possible to perform any statistical analyses on the item. For example:

What do you think about this report? __

A. It's the worst report I've read

B. It's somewhere between the worst and best
C. It's the best report I've read

Since almost all responses would be choice B, very little information is learned. Design your
questions so they are sensitive to differences between respondents. As another example:

Are you against drug abuse? (circle: Yes or No)

Again, there would be very little variability in responses and we'd be left wondering why we
asked the question in the first place.
6. Follows comfortably from the previous question. Writing a questionnaire is similar to
writing anything else. Transitions between questions should be smooth. Grouping questions that
are similar will make the questionnaire easier to complete, and the respondent will feel more
comfortable. Questionnaires that jump from one unrelated topic to another feel disjointed and are
not likely to produce high response rates.

7. Does not presuppose a certain state of affairs. Among the most subtle mistakes in
questionnaire design are questions that make an unwarranted assumption. An example of this
type of mistake is:

Are you satisfied with your current auto insurance? (Yes or No)

This question will present a problem for someone who does not currently have auto insurance.
Write your questions so they apply to everyone. This often means simply adding an additional
response category.

Are you satisfied with your current auto insurance?

___Yes
___No
___ Don't have auto insurance

One of the most common mistaken assumptions is that the respondent knows the correct answer
to the question. Industry surveys often contain very specific questions that the respondent may
not know the answer to. For example:

What percent of your budget do you spend on direct mail advertising? ____

Very few people would know the answer to this question without looking it up, and very few
respondents will take the time and effort to look it up. If you ask a question similar to this, it is
important to understand that the responses are rough estimates and there is a strong likelihood of
error.

It is important to look at each question and decide if all respondents will be able to answer it. Be
careful not to assume anything. For example, the following question assumes the respondent
knows what Proposition 13 is about.

Are you in favor of Proposition 13 ?

___Yes
___No
___ Undecided
If there is any possibility that the respondent may not know the answer to your question, include
a "don't know" response category.

8. Does not imply a desired answer. The wording of a question is extremely important. We are
striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent
into giving the answer we would like to receive. Leading questions are usually easily spotted
because they use negative phraseology. As examples:

Wouldn't you like to receive our free brochure?

Don't you think the Congress is spending too much money?

9. Does not use emotionally loaded or vaguely defined words. This is one of the areas
overlooked by both beginners and experienced researchers. Quantifying adjectives (e.g., most,
least, majority) are frequently used in questions. It is important to understand that these
adjectives mean different things to different people.

10. Does not use unfamiliar words or abbreviations. Remember who your audience is and
write your questionnaire for them. Do not use uncommon words or compound sentences. Write
short sentences. Abbreviations are okay if you are absolutely certain that every single respondent
will understand their meanings. If there is any doubt at all, do not use the abbreviation. The
following question might be okay if all the respondents are accountants, but it would not be a
good question for the general public.

What was your AGI last year? ______

11. Is not dependent on responses to previous questions. Branching in written questionnaires

should be avoided. While branching can be used as an effective probing technique in telephone
and face-to-face interviews, it should not be used in written questionnaires because it sometimes
confuses respondents. An example of branching is:

1. Do you currently have a life insurance policy ? (Yes or No)

If no, go to question 3

2. How much is your annual life insurance premium ? _________

These questions could easily be rewritten as one question that applies to everyone:

1. How much did you spend last year for life insurance ? ______ (write 0 if none)

12. Does not ask the respondent to order or rank a series of more than five items. Questions
asking respondents to rank items by importance should be avoided. This becomes
increasingly difficult as the number of items increases, and the answers become less reliable.
This becomes especially problematic when asking respondents to assign a percentage to a
series of items. In order to successfully complete this task, the respondent must mentally
continue to re-adjust his answers until they total one hundred percent. Limiting the number of
items to five will make it easier for the respondent to answer.