Questionnaire Design and Scale Development: Naresh K. Malhotra, Georgia Institute of Technology
Questionnaire Design and Scale Development: Naresh K. Malhotra, Georgia Institute of Technology
Questionnaire Design and Scale Development: Naresh K. Malhotra, Georgia Institute of Technology
Introduction
This chapter describes the importance of a questionnaire and presents the process for developing questionnaires and observational forms. Guidelines for questionnaire construction are provided at each stage of the process. In addition, commonly used scaling techniques and use of questionnaires in experimentation are discussed and the design of observational forms is presented.1
What Is a Questionnaire?
A questionnaire is a formalized set of questions for obtaining information from respondents. The overriding objective is to translate the researchers information needs into a set of specific questions that respondents are willing and able to answer. While this may seem straightforward, questions may yield very different and unanticipated responses. For example, how would you answer the following question: "Which State is larger, California or Texas?" Would you answer based on population or area?
176
wording and order. A questionnaire ensures standardization and comparability of the data across interviewers, increases speed and accuracy of recording, and facilitates data processing.
177
177
Do you think JC Penney offers better variety and prices than other department stores stores?'' (Incorrect)
A yes answer will presumably be clear, but what if the answer is no? Does this mean that the respondent thinks that JC Penney does not offer better variety, that it does not offer better prices, or that it neither offers better variety nor better prices? Such a question is called a doublebarreled question because two or more questions are combined into one. To avoid confusion, these questions should be asked separately,
178
178
Do you think JC Penney offers better variety than other department stores?'' Do you think JC Penney offers better prices than other department stores?'' (Correct)
179
of interest. The aided recall approach would list a number of cereal brands and then ask, Which of these brands were advertised last night on TV? One of the risks of presenting cues is that they may bias responses and make a respondent unduly sensitive to a topic, thus distorting their answers.
Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store. (Incorrect)
In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store. 1. 2. Women's dresses Men's apparel ____ ____
180
180
3. 4. . . . 16. 17.
____ ____
The second option is preferable because it requires less effort from respondents.
Information of a personal or highly sensitive nature may be difficult to obtain from respondents. Examples of sensitive topics include money, family life, political and religious beliefs, and involvement in accidents or crimes. The respondents may be embarrassed to answer such questions because accurate responses may threaten their prestige or self-image. To increase the likelihood of obtaining sensitive information, such topics should be placed at the end of the questionnaire. By then, rapport has been created and legitimacy of the project established, making respondents more willing to give information. Where appropriate, sensitive information should be obtained in the form of response categories rather than asking for specific figures. While respondents may refuse to answer the question
181
They may be willing to check the appropriate income category. A better way of obtaining information on income is to ask:
Which one of the following categories best describes your households annual income? under $25,000 __ __ __ $25,001-$50,000 $50,001-$75,000 over $75,000
Unstructured Questions
Unstructured questions are open-ended questions that respondents answer in their own words. They are also referred to as free-response or free-answer questions. Open-ended questions are good as first questions on a topic. They enable the respondents to express general attitudes and opinions that can help the researcher interpret their responses to structured questions. Open-ended questions allow the respondent to express their attitudes or opinions without the bias associated with restricting responses to predefined alternatives. Thus, they can be useful in identifying underlying, motivations, beliefs, and attitudes. Analysis of the verbatim
182
182
comments provides a rich context for interpreting later questions. Unstructured questions are useful in exploratory research. The disadvantages of unstructured questions relate to recording error, data coding, and the added complexity of analysis. In personal or telephone interviews, successfully recording verbatim comments depends entirely on the recording skills of the interviewer. Interviewer bias is introduced as decisions are made regarding whether to record answers verbatim or write down only the main points. Tape recorders should be used if verbatim reporting is important. In general, open-ended questions are useful in exploratory research and as opening questions. However, in a large survey, the complexity of recording, tabulation, and analysis outweighs their advantages.
Structured Questions
Structured questions specify the set of responses as well as their format. A structured question may offer multiple-choices, or a scale.
Multiple-Choice Questions
In multiple-choice questions, the researcher provides a choice of answers, and respondents are asked to select one or more of the alternatives given. Consider the following question: Do you intend to travel overseas within the next six months? _____ Definitely will not travel _____ Probably will not travel _____ Undecided _____ Probably will travel _____ Definitely will travel
183
183
Two concerns in designing multiple-choice questions are (1) the number of alternatives that should be included and (2) order or position bias. Multiple choice questions should include choices that cover the full range of possible alternatives. The alternatives should be mutually exclusive and collectively exhaustive. An other (please specify) category should be included where appropriate. Instructions should clearly indicate whether the respondent is to choose only one alternative or select all that apply. (For example, Please indicate all the brands of cereals that you have consumed in the past week.) As the list of choices increases, the questions become more difficult to answer. When the alternative list becomes long, the researcher should consider using more than one question to simplify the workload for respondents. Order or position bias is the respondents' tendency to check an alternative merely because it occupies a certain position in a list. Alternatives that appear at the beginning and, to a lesser degree, at the end of a list have a tendency to be selected most often. When questions relate to numeric values (quantities or prices), there is a tendency to select the central value on the list. Order bias can be controlled by preparing several forms of the questionnaire with changes in the order of the alternatives from form to form. Unless the alternatives represent ordered categories, each alternative should appear once in each of the extreme positions, once in the middle, and once somewhere in between. Multiple choice questions are easier for respondents to answer. They are also easier to analyze and tabulate than open-ended questions. Interviewer bias is also reduced, given that these types of questions work very well in self-administered conditions. Respondent cooperation in general is improved if the majority of the questions are structured. Multiple-choice questions are not without disadvantages. It is difficult to develop effective multiple-choice options. Often, exploratory research must be conducted using open184
184
ended questions to identify the appropriate response options. When large numbers of respondents check the other (please specify) category, it indicates that the alternative list may be seriously flawed. The list of options itself also introduces bias. Given their importance, scales are discussed in the next section.
Likert Scale
Named after its developer, Rensis Likert, the Likert scale is one of the most widely used itemized scales. The end-points of a Likert scale are typically strongly disagree and strongly agree. The respondents are asked to indicate their degree of agreement by checking one of five response categories. The following example shows how a Likert scale was used in a retailing study.
Strongly disagree
Disagree
2X
185
1 1
2X 2
3 3X
4 4
5 5
The data are typically treated as interval scale. When using this approach to determine the total score for each respondent on each store, it is important to use a consistent scoring procedure so that a high (or low) score consistently reflects a favorable response. This requires that the categories assigned to the negative statements by the respondents be scored by reversing the scale. Note that for a negative statement, an agreement reflects an unfavorable response, whereas for a positive statement, agreement represents a favorable response. Accordingly, a strongly agree response to a favorable statement and a strongly disagree response to an unfavorable statement would both receive scores of five.
In the scale shown above, if a higher score is to denote a more favorable attitude, the scoring of item 2 will be reversed while analyzing the data. Each respondent's total score for each store is calculated. A respondent will have the most favorable attitude toward the store with the highest score. The Likert scale has several advantages. It is easy for the researcher to construct and administer this scale, and it is easy for the respondent to understand. Therefore, it is suitable for mail, telephone, personal, or electronic interviews. Several variants of the Likert scale are commonly used in marketing that vary the number of scale points (for example, 7 or 9 points) as well as the descriptors (for example, importance, familiarity) and other characteristics. The major disadvantage of the Likert scale is that it takes longer to complete than other itemized rating scales. Respondents have to read the entire statement rather than a short phrase.
186
186
Likert Scale Decisions The researcher must make six major decisions when constructing Likert type of scales. 1. The number of scale categories to use 2. Balanced versus unbalanced scale 3. Odd or even number of categories 4. Forced versus nonforced choice
187
number of categories and a neutral point. The decision to use an odd or even number of categories depends on whether some of the respondents may be neutral on the response being measured. If a neutral or indifferent response is possible from at least some of the respondents, an odd number of categories should be used.
188
On the surface, this may seem to be a well-defined question, but we may reach a different conclusion when we examine it under the microscope of who, what, when, and where.
Defining the Question The Respondent It is not clear whether this question relates to the individual respondent or the respondent's total household.
What
The Brand of Bath soap It is unclear how the respondent is to answer this question if more than one brand is used.
189
189
When
Unclear The time frame is not specified in this question. The respondent could interpret it as meaning the bath soap used this morning, this week or over the past year.
Where
A more clearly defined question might read: Which brand or brands of bath soap have you personally used at home during the past month? In case of more than one brand, please list all the brands that apply. (Correct)
190
Are snack foods readily available when you want to buy them? (Correct)
In a typical month, how often do you go to a movie theater to see a movie? _____ Never _____ Occasionally _____ Sometimes _____ Often _____ Regularly (Incorrect)
The categories of this multiple choice question can have different meanings to different people, leading to response bias. Three respondents who go to movie theaters once a month may check three different categories: occasionally, sometimes, and often. The following is a much better worded question:
191
191
In a typical month, how often do you go to a movie theater to see a movie? _____ Less than once _____ 1 or 2 times _____ 3 or 4 times _____ More than 4 times (Correct)
This question is less ambiguous because each respondent is answering it from a consistent frame of reference. Response categories have been objectively defined, and respondents are no longer free to interpret them in their own way.
192
Do you think that America should provide financial aid to poor foreign countries? _____ Yes _____ No _____ Don't know (Correct) Words can lead respondents in a particular direction. Identification of the research sponsor can have the same effect. When respondents are made aware of the sponsor, they tend to answer questions about that sponsor in a positive manner. Likewise, the mention of a prestigious or nonprestigious name can bias the response, as in, Do you agree with the American Dental Association that Colgate is effective in preventing cavities?
193
193
When arranging questions in a proper order, the researcher should consider the opening questions, the type of information sought, difficult questions, and the effect on subsequent questions. Questions should be arranged in a logical order, organized around topic areas. Opening questions set the stage for the remainder of the questionnaire. They serve a variety of purposes. They can introduce the topic, attempt to gain the confidence and cooperation of respondents, or establish the legitimacy of the study. The opening questions should be interesting, simple, and nonthreatening. Questions that ask respondents for their opinions are always good openers because most people like to express their opinions. Some studies require a prescreening of the respondents to ensure that they are eligible to participate in the interview. In these cases, qualifying questions are used as opening questions. Three types of information are obtained from a questionnaire: (1) basic information, (2) classification information, and (3) identification information. Basic information relates directly to the research problem. Classification information consists of socioeconomic and demographic characteristics. It is used to classify the respondents in order to analyze results across different groups. Identification information includes name, address, and telephone number. Identification information may be obtained for a variety of purposes, including verifying that the respondents listed were actually interviewed, remitting promised incentives, and so on. Because basic information is the most important aspect of a study, it should be obtained first, followed by classification and then identification information. Classification and identification information is of a more personal nature. Respondents may resist answering a series of personal questions. Therefore, these types of questions should appear at the end of the questionnaire. Questions that could be perceived as difficult should be placed late in the sequence after a relationship has been established and the respondent is involved in the process. The last
194
194
question of the classification section is typically income information; the respondent's telephone number is the final item in the identification section for the same reasons. Initial questions can influence questions asked later in a questionnaire. As a rule, a series of questions should start with a general introduction to a topic, followed by specific questions related to the topic. This prevents specific questions from biasing responses to the general questions. Going from general to specific is called the funnel approach, because you begin with broader (more general) questions and then ask narrower (more specific) questions, reflecting the shape of a funnel.2 Although the funnel approach is more commonly used, sometimes the inverted funnel approach may be used when the respondents do not have clearly formulated views about a topic or when they lack a common frame of reference in responding to general questions on the topic. In this approach, the specific questions are asked first followed by more general questions. Questions should be asked in a logical order, organized around topic areas. When switching topics, brief transitional phrases or sentences should be used to help respondents switch their train of thought, for example, In this section, we ask questions related to your purchase of a new car in the last six months.
195
be similar to the respondents in terms of their background characteristics, familiarity with the topic, and attitudes and behaviors of interest (see Diamantopoulos, Schlegelmilch, & Reynolds, 1994; Martin & Polivka, 1995). Pretests are best done by personal interviews, even if the actual survey is to be conducted by telephone, mail, or electronically, so that interviewers can observe respondent reactions and attitudes. After the necessary changes have been made, another pretest could be administered using the actual data collection approach, if it is mail, telephone, or electronic. This stage of the pretest will reveal any potential problems in the interviewing method to be used in the actual survey. The pretest should be conducted in an environment and context similar to that of the actual survey. Based on feedback from the pretest, the questionnaire should be edited, and the identified problems corrected. After each significant revision of the questionnaire, another pretest should be conducted, using a different sample of respondents. Pretesting should be continued until no further changes are needed. As a final step, the responses obtained during the pretest should be coded and analyzed. The analysis of pretest responses can serve as a check on the adequacy of the problem definition, and provide insight into the nature of the data as well as analytic techniques that will be required.
196
Individual Question Content 1. 2. Is the question necessary? Are several questions needed instead of one to obtain the required information in an unambiguous manner? 3. Do not use double-barreled questions.
197
197
2.
If respondents are not likely to be informed, filter questions that measure familiarity, product use, and past experience should be asked before questions about the topics themselves.
3. 4.
Can the respondent remember? Questions that do not provide the respondent with cues can underestimate the actual occurrence of an event.
5. 6. 7. 8.
Can the respondent articulate? Minimize the effort required of the respondents. Make the request for information seem legitimate. Is the information sensitive?
Choosing Question Structure 1. 2. 3. Open-ended questions are useful in exploratory research and as opening questions. Use structured questions whenever possible. In multiple-choice questions, the response alternatives should include the set of all possible choices and should be mutually exclusive. 4. In a dichotomous question, if a substantial proportion of the respondents can be expected to be neutral, include a neutral alternative. 5. Consider the use of the split ballot technique to reduce order bias in dichotomous and multiple-choice questions. 6. If the response alternatives are numerous, consider using more than one question.
Choosing Question Wording 1. 2. Define the issue in terms of who, what, when, where, why, and way (the six Ws). Use ordinary words. Words should match the vocabulary level of the respondents.
198
198
3.
Avoid ambiguous words: usually, normally, frequently, often, regularly, occasionally, sometimes, and so on.
4. 5. 6. 7. 8.
Avoid leading questions that clue the respondent to what the answer should be. Avoid implicit alternatives that are not explicitly expressed in the options. Avoid implicit assumptions. Respondent should not have to make generalizations or compute estimates. Use positive and negative statements.
Determine the Order of Questions 1. 2. 3. The opening questions should be interesting, simple, and nonthreatening. Qualifying questions should serve as the opening questions. Basic information should be obtained first, followed by classification, and, finally, identification information. 4. 5. 6. Difficult, sensitive, or complex questions should be placed late in the sequence. General questions should precede specific questions. Questions should be asked in a logical order.
Pretesting 1. 2. Always pretest Test all aspects of the questionnaire, including question content, wording, sequence, form and layout, question difficulty, and instructions. 3. Use respondents in the pretest that are similar to those who will be included in the actual survey. 4. 5. Begin the pretest by using personal interviews. Conduct the pretest by mail, telephone, or electronically if those methods are to be used in the actual survey.
199
199
6. 7.
Use a variety of interviewers for pretests. The pretest sample size should be small, varying from 15 to 30 respondents for the initial testing.
8.
After each significant revision of the questionnaire, conduct another pretest, using a different sample of respondents.
9.
Summary
To collect quantitative primary data, a researcher must design a questionnaire or an observational form. A questionnaire must translate the information needed into a set of specific questions. Designing a questionnaire is as much an art as it is a science. We can provide guidelines for development, but no one optimal questionnaire design fits every research need. The process begins by specifying the information needed. The next step is to decide on the content of individual questions. Questions must be written to overcome the respondents' inability to answer. Respondents may be unable to answer if they are not informed or cannot remember. When too much effort is required, respondents will be unwilling to participate. Questions that attempt to collect sensitive information may also be met with resistance. Questions can be unstructured (open-ended) or structured to a varying degree. Structured questions include multiple-choice questions and scales. The commonly used scales are Likert, semantic differential and Stapel. Determining the wording of each question involves defining the issue, using ordinary words, using unambiguous words, and using dual statements. The issue should be clearly defined in terms of who, what, when, and where. The researcher should avoid leading questions. Once the questions have been worded, the order in which they will appear in the questionnaire
200
200
must be decided. Special consideration should be given to opening questions, type of information, difficult questions, and the effect on subsequent questions. The questions should be arranged in a logical order. The effectiveness of all these design decisions must be assessed in a pretest. Questionnaire design is a very important aspect of marketing research and effort devoted to this task can payoff handsomely in terms of the quality and completeness of the data and the results.
ENDNOTES
1
The material presented here is drawn from the work of Malhotra (2004) and Malhotra and
Peterson (2006). For more information, including detailed references, please consult these books.
Rating a brand on specific attributes early in a survey may affect responses to a later overall
brand evaluation (for more information on this, see Bartels, 2002; Bickart, 1993).
201
201
REFERENCES Bartels, L. M. (2002). Question order and declining faith in elections. Public Opinion Quarterly, 66(1), 6779. Bickart, B. A. (1993). Carryover and backfire effects in marketing research. Journal of Marketing Research, 30, 5262. Diamantopoulos, A., Schlegelmilch, B. B., & Reynolds, N. (1994). Pretesting in questionnaire design: The impact of respondent characteristics on error detection. Journal of the Market Research Society, 36(October), 295314. Malhotra, N. K. (2004). Marketing research: An applied orientation (4th ed.). Upper Saddle River, NJ: Prentice Hall -------, & Peterson, M. (2006). Basic marketing research: A decision making approach (2d ed.). Upper Saddle River, NJ: Prentice Hall. Martin, E. & Polivka, A. E. (1995). Diagnostics for redesigning survey questionnaires. Public Opinion Quarterly, 59(4), 547567.
202
202