Education

How to design a questionnaire

Wai-Ching Leung has some practical advice on questionnaires
As discussed in a previous issue a survey involves directly collecting information from people (or sometimes organisations) whom we are interested in.1 The types of information will take account of the peoples or organisations level of knowledge, attitude, personalities, beliefs, or preferences. Questionnaires are widely used to collect such information. Well designed questionnaires are highly structured to allow the same types of information to be collected from a large number of people in the same way and for data to be analysed quantitatively and systematically. Questionnaires are best used for collecting factual data and appropriate questionnaire design is essential to ensure that we obtain valid responses to our questions.

Objectives in designing questionnaires

There are two main objectives in designing a questionnaire: q To maximise the proportion of subjects answering our questionnairethat is, the response rate. q To obtain accurate relevant information for our survey. To maximise our response rate, we have to consider carefully how we administer the questionnaire, establish rapport, explain the purpose of the survey, and remind those who have not responded. The length of the questionnaire should be appropriate. In order to obtain accurate relevant information, we have to give some thought to what questions we ask, how we ask them, the order we ask them in, and the general layout of the questionnaire.

dependent factors include the students level of relevant knowledge, skills, and attitudes. The independent factors might include students learning styles, GCSE and A level grades, socioeconomic status, ethnicity, etc. Confounding variables might include the types and quality of teaching in each medical school. Sometimes, additional questions are used to detect the consistency of the subjects responses. For example, there may

be a tendency for some to tick either agree or disagree to all the questions. Additional contradictory statements may be used to detect such tendencies.

Wording of individual questions

The way questions are phrased is important and there are some general rules for constructing good questions in a questionnaire.

Use short and simple sentences

Box 1: Advantages of open or closed format
Open format
Allows exploration of the range of possible themes arising from an issue Can be used even if a comprehensive range of alternative choices cannot be compiled

Deciding what to ask

As discussed in last months issue, there are three potential types of information: q Information we are primarily interested inthat is, dependent variables. q Information which might explain the dependent variablesthat is, independent variables. q Other factors related to both dependent and independent factors which may distort the results and have to be adjusted forthat is, confounding variables. Let us take as an example a national survey to find out students factors predicting the level of certain knowledge, skills, and attitudes at the end of their undergraduate medical course. The
STUDENT BMJ VOLUME 9 JUNE 2001 studentbmj.com

Short, simple sentences are generally less confusing and ambiguous than long, complex ones. As a rule of thumb, most sentences should contain one or two clauses. Sentences with more than three clauses should be rephrased.

Ask for only one piece of information at a time

For example, Please rate the lecture in terms of its content and presentation asks for two pieces of information at the same time. It should be divided into two parts: Please rate the lecture in terms of (a) its content, (b) its presentation.

Closedthat is, forced choiceformat

Easy and quick to fill in Minimise discrimination against the less literate (in self administered questionnaire) or the less articulate (in interview questionnaire) Easy to code, record, and analyse results quantitatively Easy to report results

Avoid negatives if possible

Negatives should be used only sparingly. For example, instead of asking students whether they agree with the statement,
187

Education

Small group teaching should not be abolished, the statement should be rephrased as, Small group teaching should continue. Double negatives should always be avoided.

Ask precise questions

Questions may be ambiguous because a word or term may have a different meaning. For example, if we ask students to rate their interest in medicine, this term might mean general medicine (as opposed to general surgery) to some, but inclusive of all clinical specialties (as opposed to professions outside medicine) to others. Another source of ambiguity is a failure to specify a frame of reference. For example, in the question, How often did you borrow books from your library? the time reference is missing. It might be rephrased as, How many books have you borrowed from the library within the past six months altogether?

approach: Please tick one or more of the following items which correspond to how you have answered degree examination questions in the past. In the list of items, include copy from other students as one of many items. Thirdly, the everybody approach: As we all know, most medical students have copied other students answers in degree exams. Do you happen to be one of them? Fourthly, other people approach. This approach was used in the recent medical student survey.3 In this survey, students were given the scenario, John copies answers in a degree exam from Jean. They were then asked, Do you feel John is wrong, what penalty should be imposed for John, and have you done or would you consider doing the above?

Minimise bias
People tend to answer questions in a way they perceive to be socially desired or expected by the questioner and they often look for clues in the questions. Many apparently neutral questions can potentially lead to bias. For example, in the question, Within the past month, how many lectures have you missed due to your evening job? students may perceive the desired responses to be never to the first question. This question could be rephrased as, Within the past month, how many times did your evening job commitment clash with lectures? How many times did you give priority to your evening job? Take another example. The question, Please rate how useful the following text-

Box 2: Types of closed (forced choice) format

Choice of categories For example, What is your marital status? [ ] Single [ ] Married [ ] Divorced [ ] Widowed Likert style scale For example, Statistics is an interesting subject
Strongly disagree Disagree Cannot decide Agree Strongly agree

Ensure those you ask have the necessary knowledge

For example, in a survey of university lecturers on recent changes in higher education, the question, Do you agree with the recommendations in the Dearing report on higher education? is unsatisfactory for several reasons. Not only does it ask for several pieces of information at the same time as there are several recommendations in the report, the question also assumes that all lecturers know about the relevant recommendations.

Differential scales For example, How would you rate the presentation?
Extremely interesting 1 2 3 4 5 6 7 8 9 10 Extremely dull It may also be diagrammatic:

Level of details
It is important to ask for the exact level of details required. On the one hand, you might not be able to fulfil the purposes of the survey if you omit to ask essential details. On the other hand, it is important to avoid unnecessary details. People are less inclined to complete long questionnaires. This is particularly important for confidential sensitive information, such as personal financial matters or marital relationship issues.
Extremely interesting Extremely dull

Checklists For example, Circle the clinical specialties you are particularly interested in
General medicine General surgery Ophthalmology Paediatrics Obstetrics and gynaecology Orthopaedics Accident and emergency General practice

Sensitive issues
It is often difficult to obtain truthful answers to sensitive questions. Clearly, the question, Have you ever copied other students answers in a degree exam? is likely to produce either no response or negative responses. Less direct approaches have been suggested.2 Firstly, the casual approach: By the way, do you happen to have copied other students answers in a degree exam? may be used as a last part of another decoy question. Secondly, the numbered card
188

Ranking For example, Please rank your interests in the following specialties (1= most interesting, 8=least interesting)
General medicine General surgery Ophthalmology Paediatrics Obstetrics and gynaecology Orthopaedics Accident and emergency General practice

STUDENT BMJ VOLUME 9

JUNE 2001 studentbmj.com

Education

books are. Please also state whether they are included in your lecturers recommended reading list? There is a risk that the students may perceive that they should rate books recommended by lecturers more favourably than those not recommended by their lecturers. This risk may be minimised by putting the second question later on in the questionnaire.

every question. Filter questions are useful to ensure that respondents answer only relevant questions. However, avoid a highly complex filter regime. Make good use of arrows and boxes to clarify the filter regime.

Introduction, personalised letter, and ending

It seems a good idea to have either a personalised covering letter or at least an introduction explaining briefly the purpose of the survey, the importance of the respondents participation, who is responsible for the survey, and a statement guaranteeing confidentiality.5 A personalised letter can be easily generated using mail-merge on a word processor. It is also important to thank the respondent at the end of the questionnaire.

Format of responses
The responses can be in open or closed formats. In an open ended question, the respondents can formulate their own answers. In closed format, respondents are forced to choose between several given options. The advantages of each of these formats are shown in box 1. It is possible to use a mixture of the two formatsfor example, give a list of options, with the final option of other followed by a space for respondents to fill in other alternatives. There are several forced choice formats. These are shown in box 2. Out of these formats, ranking is probably least frequently used, as the responses are relatively difficult to record and analyse.

How to administer the questionnaires

There are several ways of administering questionnaires. They may be self administered or read out by interviewers. Self administered questionnaires may be sent by post, email, or electronically online. Interview administered questionnaires may be by telephone or face to face. Advantages of self administered questionnaires include: q Cheap and easy to administer. q Preserve confidentiality. q Can be completed at respondents convenience. q Can be administered in a standard manner.

Length of questionnaire
There are no universal agreements about the optimal length of questionnaires. It probably depends on the type of respondents. However, short simple questionnaires usually attract higher response rates than long complex ones. In a BMJ survey of stroke survivors both the response rate and the proportion of completed forms were higher for a shorter questionnaire (six questions with a visual analogue scale) compared with a longer and more complex questionnaire (with 34 questions).4

might ask each respondent in great detail about a limited number of questions: effects of different wordings, what they have in mind when they give a particular answer, how they understand a particular word, etc. In the second phase the whole questionnaire is administered by interviewers. Analysis of the responses and the interviewers comments are used to improve the questionnaire. Ideally, there should be sufficient variations in responses among respondents; each question should measure different qualitiesthat is, the responses between any two items should not be very strongly correlated and the non-response rate should be low. In the third phase the pilot test is polished to improve the question order, filter questions, and layout.

Conclusions
Questionnaires must be carefully designed to yield valid information. Meticulous attention must be paid to ensure that individual questions are relevant, appropriate, intelligible, precise, and unbiased. The order of the questions must be carefully arranged, and the layout of the questionnaire must be clear. It is wise to draft a clear personalised covering letter. Questionnaires must first be piloted and evaluated before the actual survey.

Advantages of interview administered questionnaires include:

Allow participation by illiterate people. Allow clarification of ambiguity. The exact method of administration also depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews.
q q

Arranging the questions

The order of the questions is also important. Some general rules are: q Go from general to particular. q Go from easy to difficult. q Go from factual to abstract. q Start with closed format questions. q Start with questions relevant to the main subject. q Do not start with demographic and personal questions. It is useful to use a variety of question format as shown in box 2 to maintain the respondents interest. When a series of semantic differential scales are used, it may be a good idea to mix positive negativefor example, interesting to dullwith negative positivefor example, useless to usefulscales. This might make the respondents think more and avoid the tendency to tick the same response for
STUDENT BMJ VOLUME 9 JUNE 2001 studentbmj.com

Further reading:
Abramson JH, Abramson ZH. Survey methods in community medicine. Edinburgh: Churchill Livingstone, 1999. Bowling A. Research methods in health: investigating health and health services. Buckingham: Open University Press, 1997.

Piloting and evaluation of questionnaires

Given the complexity of designing a questionnaire, it is impossible even for the experts to get it right the first time round. Questionnaires must be pretestedthat is, pilotedon a small sample of people characteristic of those in the survey. In a small survey, there might be only pretesting of the drafted questionnaire. In a large survey, there may be three phases of piloting. In the first phase we

Wai-Ching Leung lecturer in public health medicine, University of East Anglia

w-c.leung@uea.ac.uk 1 2 3 Leung WC. How to conduct a survey. StudentBMJ 2001;9:143-5. (May.) Barton JA. Asking the embarrassing question. Public Opinion Quarterly 1958;22:67-8. Rennie SC, Crosby JR. Are tomorrows doctors honest? Questionnaire study exploring medical students attitudes and reported behaviour in academic misconduct. studentBMJ 2001;9:67-8.(March.) Dorman PJ, Slattery J, Farrell B, Dennis MS, Sandercock PAG. A randomised comparison of the EuroQol and Short Form-36 after stroke. BMJ 1997;315:461. Bissett AF. Designing a questionnaire: Send a personal covering letter. BMJ 1994;308:202-3.

189