Standardized Questionnaire and rating scales as data collection method

Questionnaires, inventories and scales

A questionnaire is a set of question for gathering information from individuals. Questionnaires

can be administered by mail, telephone, using face to face interview, or electronically ( i.e, by e
mail or through web based questionnaires)

When questionnaires should be used?

I. When the resources are limited and the researcher needs data from many people
II. To gather data about knowledge, beliefs, attitudes and behaviours
III. When it is important to protect the privacy of the participants

Questionnaires provide a relatively cheap, quick and efficient way of obtaining large amounts of
information from a large sample of people. Data can be collected relatively quickly because the
researcher would not need to be present when the questionnaires were completed. This is useful
for large populations when interviews would be impractical.

However, a problem with questionnaire is that respondents may lie due to social desirability.
Most people want to present a positive image of themselves and so may lie or bend the truth to
look good, e.g. pupils would exaggerate revision duration.

Also the language of a questionnaire should be appropriate to the vocabulary of the group of
people being studied. For example, the researcher must change the language of questions to
match the social background of respondents' age / educational level / social class / ethnicity etc.

Questionnaires can be an effective means of measuring the behaviour, attitudes, preferences,

opinions and intentions of relatively large numbers of subjects more cheaply and quickly than
other methods. An important distinction should be made between open-ended and closed
questions. Often a questionnaire uses both open and closed questions to collect data. This is
beneficial as it means both quantitative and qualitative data can be obtained.

Types of questions

1. Open Questions

Open questions allow people to express what they think in their own words. Open ended
questions provide a rich source of qualitative information (i.e. more descriptive than numerical)
as there is no restriction to the response. However, they are harder to analyse and make
comparisons from.
If you want to gather more in-depth answers from your respondents, then open questions will
work better.

2. Closed Questions

Closed questions structure the answer by allowing only answers which fit into categories that
have been decided in advanced by the researcher. Data that can be placed into a category is
called nominal data.

The options can be restricted to as few as two (e.g. 'yes' or 'no', 'male' or 'female'), or include
quite complex lists of alternatives from which the respondent can choose.

The respondent provides information which can be easily converted into quantitative data (e.g.
count the number of 'yes' or 'no' answers).

Closed questions can also provide ordinal data (which can be ranked). This often involves using
a rating scale to measure the strength of an attitudes or emotions. Various kinds of rating scales
have been developed to measure attitudes directly. The most widely used is the Likert Scale.
Likert-type or frequency scales use fixed choice response formats. Respondents may be offered a
choice of five to seven or even nine pre-coded responses

Examples:

Strongly Agree, Agree, Undecided, Disagree, Strongly Disagree

Very Frequently, Frequently, Occasionally, Rarely, Never

Designing a Questionnaire

There are a number of important factors in questionnaire design.

I. Aims: Make sure that any questions asked address the aims of the research.
II. Length: The longer the questionnaire the less likely people will complete it. Questions
should be short, clear, and be to the point; any unnecessary questionnaires should be
omitted. Two sides of A4 is usually an ideal length.
III. Pilot Study: Run a small scale practice study to ensure people understand the questions.
People will also be able to give detailed honest feedback on the questionnaire design.
IV. Terminology: There should be a minimum of technical jargon.
V. Question Formation: Questions should be simple, to the point and easy to understand.
VI. Presentation: Make sure it looks professional, include clear and concise instructions.
VII. Psychometric properties: A standardized questionnaire should have adequate reliability
and validity.
Benefits of a Pilot Study

A pilot study is a practice / small-scale study conducted before the main study.

It allows the researcher to try out the study with a few participants so that adjustments can be
made before the main study, so saving time and money. It is important to conduct a questionnaire
pilot study for the following reasons:

• Check that respondents understand the terminology used in the questionnaire

• Check that emotive questions have not been used as they make people defensive and
could invalidate their answers.
• Check that leading questions have not been used as they could bias the respondent's
answer
• Ensure the questionnaire can be completed in an appropriate time frame (i.e. it's not too
long).

Psychometric properties

1. Reliability

Reliability in the context of research, particularly in the questionnaire method, refers to the
consistency and stability of measurement over time and across different conditions. It assesses
the extent to which a research instrument, such as a questionnaire, produces consistent and
dependable results. In other words, a reliable questionnaire should yield similar responses when
administered multiple times to the same group or when different people answer the same
questions under similar conditions.

There are several types of reliability that researchers can use to evaluate the reliability of a
questionnaire:

a. Test-Retest Reliability: This type of reliability assesses the consistency of questionnaire

responses over time. To measure test-retest reliability, the same questionnaire is
administered to the same group of participants on two separate occasions, with a
significant time interval in between. The responses from the two administrations should
be highly correlated if the questionnaire is reliable.
b. Internal Consistency Reliability: This type of reliability assesses the consistency of
responses within a single questionnaire. It is commonly measured using methods like
Cronbach's alpha or Kuder-Richardson formula (KR-20). These methods calculate the
degree to which the items within the questionnaire are correlated. A high internal
consistency indicates that the items in the questionnaire are measuring the same
underlying construct.
2. Validity:
Validity means that a tool measures what it sets out to measure – for example, that a pain
assessment tool measures pain intensity rather than anxiety. There are several measures of
validity that provide evidence of the quality of a study. Measures that are used to assess the
validity of data collection tools are summarized below.

a. Content validity

Whether a tool appears to others to be measuring what it says it does. Face validity is a
simple form of content validity – the researcher asks a few people to check the tool covers all
areas. A more rigorous way to assess content validity is to ask recognised experts in the area
to give their opinion on the validity of the tool.

b. Criterion validity

Concurrent or predictive validity are both measures of criterion validity. Concurrent validity
uses an already existing and well-accepted measure against which the new measure can be
compared – for example, if you were developing a new pain assessment tool you would
compare the ratings obtained from the new tools with those obtained using a previously
validated tool. Predictive validity measures the extent to which a tool can predict a future
event of interest – for example, does a tool developed to measure the risk of pressure sores in
children in hospital in fact identify the children at risk? Criterion validity is usually measured
using a correlation coefficient – when the correlation is high, the tool can be considered
valid.

c. Construct validity

This tests the link between a measure and the underlying theory. If a test has construct
validity, you would expect to see a reasonable correlation with tests measuring related areas.
Evidence of construct validity can be provided by comparing the results obtained with the
results obtained using other tests, other(related) characteristics of the individual or factors in
the individual’senvironment which would be expected to affect test performance.
Constructvalidity is usually measured using a correlation coefficient – when the correlationis
high, the tool can be considered valid.

The advantages of questionnaires

• Practical
• Large amounts of information can be collected from a large number of people in a short
period of time and in a relatively cost effective way
• Can be carried out by the researcher or by any number of people with limited affect to its
validity and reliability
• The results of the questionnaires can usually be quickly and easily quantified by either a
researcher or through the use of a software package
 • Can be analysed more 'scientifically' and objectively than other forms of research
• When data has been quantified, it can be used to compare and contrast other research and
may be used to measure change
• Positivists believe that quantitative data can be used to create new theories and / or test
existing hypotheses

The disadvantages of questionnaires

• Is argued to be inadequate to understand some forms of information - i.e. changes of

emotions, behaviour, feelings etc.
• Phenomenologists state that quantitative research is simply an artificial creation by the
researcher, as it is asking only a limited amount of information without explanation
• There is no way to tell how truthful a respondent is being
• There is no way of telling how much thought a respondent has put in
• The respondent may be forgetful or not thinking within the full context of the situation
• People may read differently into each question and therefore reply based on their own
interpretation of the question - i.e. what is 'good' to someone may be 'poor' to someone
else, therefore there is a level of subjectivity that is not acknowledged