Clinical Governance Advice

No. 5
October 2003

UNDERSTANDING AUDIT

1. Background
Clinical governance provides a framework for accountability and quality improvement. While
research is concerned with discovering the right thing to do, audit is concerned with ensuring that
the right thing is done.1 A First class service 2 outlined structures within the National Health
Service (NHS) for setting standards: the National Institute of Clinical Excellence (NICE) and the
National Service Frameworks (NSF); and monitoring performance: the Commission for Health
Improvement (CHI) and the Performance Assessment Framework (PAF). Analogous mechanisms
have been established in Scotland.2 Modifications have been under way in response to the Kennedy
report,3 which recommended bringing together national audit and assessment activity within a
single independent organisation, the Commission for Healthcare Audit and Inspection (CHAI) in
April 2004.4 This organisation will provide external audit and quality assurance mechanisms for
the NHS.

National clinical audit funding will cover central costs (e.g. database design, tools for data
collection, transmission and feedback), data analysis, ‘benchmarking’, project management and
administration. This funding will not cover local data collection or resource implications of
change. However, as good medical practice, individual doctors are required to undertake clinical
audit.5 NHS trusts are required to ensure that hospital doctors take part in national clinical audits,
confidential enquiries and audit programmes endorsed by the CHI. Obstetricians and
gynaecologists therefore need to understand the principles of clinical audit. In this document, the
issues surrounding audit methodology, organisation and implementation of change are discussed.

2. Definition
Clinical audit is a quality improvement process that seeks to improve patient care and outcomes
through systematic review of care against explicit criteria and the implementation of change.
Aspects of the structure, processes and outcomes of care are selected and systematically evaluated
against explicit criteria. Where indicated, changes are implemented at an individual, team or
service level and further monitoring is used to confirm improvement in healthcare delivery.6

3. Evidence
A review of the evidence by NICE concluded that audit is an effective method for improving the
quality of care. The same review also described the audit methods associated with successful audit
projects.6 These findings are drawn upon in this document to give practical advice for under-
taking audit.

Clinical Governance Advice No. 5 1 of 10

4. What can be audited?
Audit may evaluate the structure (organisation or provision) of services, the process of care or the
outcome of care against an agreed standard.

4.1 Measure of structure or service provision

Audit can provide an overview of service provision. For example, research evidence shows
that the outcome for patients with ovarian cancer is better if they are operated on by an
appropriately trained gynaecologist and managed within the framework of a
multidisciplinary team.7–9 An audit of the referral and management of patients with ovarian
cancer can provide an overview of service provision in this area. Good quality healthcare
services need to be patient-centred and acceptable to those who use them. Measuring the
views of those who use services enables healthcare providers to assess the service delivered
from the patient’s perspective.

4.2 Process measure

Process measures are clinical practices that have been evaluated in research and shown to
have an influence on outcome. For example, research evidence shows that the use of antenatal
steroids has improved perinatal outcome. Evaluation of this process of care would entail
measuring the proportion of appropriate women who received antenatal steroids. Process
measures may be used to assess the quality of care and have some advantages over outcome
measures:
● they provide a more direct measure of the quality of care provided
● they occur more frequently, so smaller samples are needed
● the findings are easier to interpret
● as smaller audits are needed, they cost less to achieve.

4.3 Outcome measure

Outcome measure is the physical or behavioural response to an intervention; for example, the
health status (dead or alive), cure following surgery for stress incontinence, level of
knowledge or satisfaction (e.g. users’ views on the care they have received). Outcomes can be
desirable; for example, improvement in the patient’s condition or quality of life, or
undesirable, e.g. adverse effects of a treatment. The assessment of outcomes such as cancer
survival rates is fundamental to measuring quality of care but the use of outcomes alone in
assessing quality of care has limitations:
● outcomes are not a direct measure of the care provided; ascribing causal factors to
variation in care may be problematic, e.g. social and health inequalities may contribute to
variation in mortality rates
● not all patients who experience substandard care will have a poor outcome
● many factors contribute to eventual outcome (e.g. disease severity, health status and social
and health inequalities); therefore, mechanisms to account for these differences are
required (e.g. case-mix adjustment for co-morbidity)
● outcomes may be delayed
● research evidence about the impact of some care processes on outcome is limited
● adverse outcomes occur less frequently so larger samples will be needed.

Despite all the difficulties associated with the interpretation of outcome measures, mortality
and morbidity measures are important and this is a major justification for regular monitoring.
‘Critical incident’ or ‘adverse event’ reporting involves the identification of patients where an
adverse event has occurred, such as the Confidential Enquiries into Maternal Deaths
(CEMD), the Confidential Enquiry into Stillbirths and Deaths in Infancy (CESDI) and the

Clinical Governance Advice No. 5 2 of 10

 National Confidential Enquiry into Perioperative Deaths (NCEPOD).10 These are examples
of outcome reporting. However, only adverse events are reported.

Guidelines Evidence

Patient and public What are we Outcomes

involvement trying to achieve?

Benchmarking Consensus
Process
redesign Sampling

Have we made Data Are we

things better? analysis achieving it?

Monitoring Questionnaire Data

design collection

Doing something to Why are we not

make things better achieving it?

Process Continuous Facilitation

redesign quality
improvement Change
management
Benchmarking

Figure 1. The Clinical audit cycle

5. The audit cycle

Audit can be considered to have five principal steps, commonly referred to as the audit cycle
(Figure 1):
● selection of a topic
● identification of an appropriate standard
● data collection to assess performance against the prespecified standard
● implementation of changes to improve care if necessary
● data collection for a second, or subsequent, time to determine whether care has improved.

Audit projects require a multidisciplinary approach with the involvement of stakeholders

(including consumers or users of the service provided) and the local audit department at the
planning stage. Good planning and resources are also necessary to ensure its success.

5.1 Selection of a topic

It is essential to establish clear aims and objectives at this stage so that the audit is focused
and addresses specific issues within the selected topic. A key consideration is ‘how will we use
the results of this audit to change or improve practice?’.

In selecting a topic for audit, priority should be given to common health concerns, areas
associated with high rates of mortality, morbidity or disability, and those where good research
evidence is available to inform practice or aspects of care that use considerable resources. It is
important to involve those who will be implementing change at this stage of the audit process.

Clinical Governance Advice No. 5 3 of 10

5.2 Identification of an appropriate standard
5.2.1 Review criteria
These are defined as ‘systematically developed statements that can be used to assess specific
healthcare decisions, services and outcomes’. In audit, review criteria are generally used for
assessing care; this approach is sometimes referred to as criterion-based audit. The criterion
is the reference point against which current practice is measured. High-quality evidence-based
guidelines can be used as the starting point for developing criteria. Where this is not possible,
criteria should be agreed by a multidisciplinary group including those involved in providing
care and those who use the service. Where criteria are based on the views of professionals or
other groups, formal consensus methods are preferable. Review criteria should be explicit
rather than implicit and need to:6
● lead to valid judgements about the quality of care, and therefore should be based on
research evidence about the importance of those aspects of care
● relate to aspects of care that are important either to patients or in terms of clinical outcome
● be measurable.
Examples of audit topics and review criteria are given in Table 1.

Table 1 Examples of audit and review criteria

Audit topic Review criteria

Induced abortion Screening for lower genital tract organisms and treatment of positive cases among
women undergoing induced abortion should be carried out to reduce post-abortion
infective morbidity
Caesarean section A thromboprophylaxis strategy should be part of the management of women delivered by
caesarean section
Hysterectomy Transcervical resection of the endometrium or endometrial ablation should be available and
offered to women with dysfunctional uterine bleeding as an alternative to hysterectomy

5.2.2 Standard and target level of performance

This is defined as ‘the percentage of events that should comply with the criterion’ (e.g. the
proportion of women undergoing induced abortion who were screened for lower genital tract
organisms, the proportion of women delivered by caesarean section who received
thromboprophylaxis, the proportion of women with dysfunctional uterine bleeding who were
offered transcervical resection of the endometrium or endometrial ablation). Information
about the levels of performance that can be achieved may be helpful when making plans for
improvement. Target levels of performance should be examined periodically. The most
common approach for setting target levels of performance is informal agreement among the
group leading the audit or among health professionals. In some settings, external standards
can be useful. However, in many audits no explicit targets are set and the aim is to improve
upon current performance.

Target levels of performance have been most used in screening programmes. For example, in
screening for cervical cancer there are quality criteria to be met, such as the proportion of
cervical smears that have endocervical cells.

The term ‘standard’ has been used to refer to different concepts, sometimes as an alternative
word for ‘clinical guidelines’ and ‘review criteria’, either with or without a stated target level
of performance and, somewhat confusingly, also to refer to the observed or desired level of
performance. However, it has been defined as ‘the percentage of events that should comply
with the criterion’ in the interests of clarity.

Clinical Governance Advice No. 5 4 of 10

 5.2.3 Benchmarking
This is the ‘process of defining a level of care set as a goal to be attained’. There is insufficient
evidence to determine whether it is necessary to set target levels of performance in audit.
However, in some audits, benchmarking techniques could help participants in audit to avoid
setting unnecessarily low or unrealistically high target levels of performance. Reference to the
levels achieved in audits undertaken by other professionals is useful. National audits may
provide data for benchmarking. For example, the National Sentinel Caesarean Section Audit
Report11 gives regional and national data for comparison on topics such as the use of regional
anaesthesia in women having caesarean section.

5.3 Data collection to assess performance against the pre-specified standard

Data collection in criterion-based audit is generally undertaken to determine the proportion
of cases where care is in accordance with the criteria. In practice, the following points need
to be considered.

5.3.1 What data items to collect?

Consideration needs to be given to which data items are needed in order to answer the audit
question. For example, if undertaking an audit on caesarean section rates, collecting
information on the number of caesarean sections alone will not give sufficient information to
measure the caesarean section rate. Data on the number of other births that took place is also
required. In general, for audit projects with clear aims, objectives and well-defined review
criteria, it is easier to identify those data items that require collection. Definitions need to be
clear so that there is no confusion about what is being collected. The definitions will depend
upon the review criterion that is being assessed. For example, if collecting data on rupture of
membranes, it may need to be specified whether this is spontaneous or artificial.

Data collectors should always be aware of their responsibilities to the Data Protection Act12
and any locally agreed guidelines. There are also nationally agreed guidelines, known as the
Caldicott Principles, a key part of clinical and information governance. The six Caldicott
Principles are:
1. Justify the purpose(s). Every proposed use or transfer of patient-identifiable information
within or from an organisation should be clearly defined and scrutinised, with continuing
uses regularly reviewed by an appropriate guardian.
2. Do not use patient-identifiable information unless it is absolutely necessary.
3. Use the minimum necessary patient-identifiable information. Where use of patient-
identifiable information is considered to be essential, each individual item of information
should be justified with the aim of reducing identifiability.
4. Access to patient-identifiable information should be on a strict need-to-know basis. Only
those individuals who need access to patient-identifiable information should have access to
it, and they should only have access to the information items that they need to see.
5. Everyone should be aware of their responsibilities. Both clinical and nonclinical staff
should be aware of their responsibilities and obligations to respect patient confidentiality.
6. Understand and comply with the law. Someone in each organisation should be responsible
for ensuring that the organisation complies with legal requirements.

Under the Data Protection Act 1998, it is an offence to collect personal details of patients
such as name, address, or other items that are potentially identifiable for the individual
without consent. There seems to be consensus that clinical audit is part of direct patient care
and therefore consent to use of data for audit can be implied through consent to treatment,

Clinical Governance Advice No. 5 5 of 10

 provided that information is given to patients that their data may be used in this way. It is
rarely acceptable to use patient identifiers, such as names and addresses, but some form of
pseudo-anonymised identifiers may be used. Audit project protocols should be submitted to
the local research and development committee and ethics committees to seek approval if
necessary. Guidance on how to do this is can be obtained from the respective bodies.

5.3.2 How to collect the data

Sources of data include:
● routinely collected data if available (e.g. birth registers); this enables repeated data
collections with the minimum of extra effort
● clinical records
● data collection through direct observation or from questionnaire surveys of staff or patients.

Routinely collected data can be used if all the data items required are available. It will be
necessary to check the definitions for data items that are used within the routine database to
ensure its usefulness for the aims of the audit. Also, the completeness and coverage of the
routine source needs to be known.

Where the data source is clinical records, training of data abstractors and use of a standard
pro forma can improve accuracy and reliability of data collection. The use of multiple sources
of data may also be helpful. However, this can also be problematic, as it will require linking
of data from different sources with common unique identifiers.

Questionnaire surveys of staff or patients are often used for data collection. There are several
validated questionnaires on a wide range of topics that may be adapted to a specific audit
project. There is also literature on developing these (see Appendix).

5.3.3 Who will collect the data?

Thought needs to be given to who will collect the data, as well as the time and resources that
will be involved. In small audit projects it may be feasible for the principal investigators to go
through clinical notes for data abstraction. However, for larger projects, e.g. a prospective
audit on induction of labour practices within a maternity unit, it may be more appropriate
for those involved in the care of the woman giving birth (e.g. midwives or obstetricians) to
fill in standard data collection sheets. Where available, audit support staff should be involved.

5.3.4 Data management

Data that are collected on paper forms are usually entered on to electronic databases or
spreadsheets such as Microsoft Access®, Epi Info® or Microsoft Excel® for cleaning and
analysis. Data entry may be done by optical character recognition (OCR) software, optical
mark readers (OMR) or manually. OCR is most accurate for questionnaire data using tick
boxes but less accurate for free text responses. The method of data entry needs to be taken
into account when designing the questionnaire or data collection sheet. For manual data
entry, accuracy is improved if double data entry is used. However, this can be a time
consuming exercise. If the facilities and resources are available, electronic collection of data
can be considered. In this case, data are entered immediately, at source, into a computer and
saved to disk. While this is quick and requires minimal storage space, it can be difficult to
handle unexpected responses. As information is entered directly into a computer it cannot be
verified or double-entered.13

Consideration also needs to be given to the coding of responses on the database. For ease of
analysis of closed questions it is generally better to have numeric codes for responses. For
example, yes/no responses can be coded to take the value 0 for no and 1 for yes. Missing data

Clinical Governance Advice No. 5 6 of 10

 will also need to be coded; for example, with the number 9. The code assigned for missing
data should be distinguished from those where the response is ‘not known’ (if this was an
option on the questionnaire).

It is advisable to incorporate consistency checks as data are being entered, in order to

minimise errors. For example, if there are two questions:
(a) How many previous pregnancies of at least 24 weeks of gestation has this woman had?
(b) How many previous caesarean sections has she had?
A consistency check will highlight entries with responses other than 0 to question (b) if the
response to question (a) is 0.

5.3.5 Data analysis

Simple statistics are often all that is required. Statistical methods are used to summarise data for
presentation in the form of summary statistics (means, medians or percentages) and graphs.14
Statistical tests are used to find out the likelihood that the data obtained has arisen by chance
and how likely it is that a real difference exists between two groups. Before data collection has
started it is essential to know what data items will be collected, whether comparisons will be
made, and the statistical methods that will be used to make these comparisons.

Data items that have categorical responses (e.g. yes/no or A/B/C/D) can be expressed as
percentages. Some data items are collected as continuous variables; for example, mother’s
age, height and weight. These can either be categorised into relevant categories and then
expressed as percentages or, if they are normally distributed, the mean and standard
deviations can be reported. These summary statistics (percentages and means) are useful for
describing the process, outcome or service provision that was measured.

Comparisons of percentages between different groups can be made using a chi-square test; t
tests can be used to compare means between two groups, assuming that these are normally
distributed. Nonparametric statistical methods can be used for data that are not normally
distributed. These comparisons are useful in order to determine whether there are any real
differences in the observed findings; for example, when comparing audit results obtained at
different time points or in different settings. In some situations a sample-size calculation may
be necessary to ensure that the audit is large enough to detect a clinically significant difference
between groups, if one exists. In this situation, it is important to consult a statistician during
the planning stages of the audit project.

These simple statistics can be easily done using Microsoft Excel spreadsheets and Microsoft
Access databases. Other useful statistical software packages include Epi Info, SAS, SPSS,
STATA and Minitab.

5.4 Implementation of changes to improve care if necessary

Data analysis and interpretation will lead to the identification of clinical areas that should be
addressed. There are many methods by which this can be done. The feedback of audit findings
is most commonly used; for example, presentation at regular audit meetings will stimulate
discussions and solutions may be agreed. The NICE review6 identified several audits in which
change in care had occurred. Simple methods were occasionally effective, for example:
● feedback of data collected
● provision of clear data, perhaps using modern information systems, supported by active teamwork
● support from the organisation for teamwork
● use of several methods together within the context of an implementation plan.

Clinical Governance Advice No. 5 7 of 10

 Change does not always occur in audit and consideration of the reasons for failure may take
place after the second data collection. Resistance to change among local professionals or in
the organisational environment or team should be considered. Patients themselves may have
preferences for care that make change difficult.

The significance of teamwork, culture and resistance to change has led several authors to
propose frameworks for planning implementation. These usually include analysis of the
barriers to change and use of theories of individual, team or organisational behaviour to
select strategies to address the barriers. For some topics, such as adverse incidents, systems
for continuous data collection may be justified.

6. Organisation of audit
The NICE review6 found that some methods of organising audit programmes were better than
others. The following features are associated with successful audit:
● structured programmes with realistic aims and objectives
● leadership and attitude of senior management
● nondirective, hands-on approach
● support of staff, strategy groups and regular discussions
● emphasis on teamworking and support
● environment conducive to conducting audit.

6.1 Common reasons why audits fail

● Failure to participate and attitudes to audit.
Involving all stakeholders (including service users) in the project can encourage
participation. It is important to recognise the attitudes of those whose behaviour is being
audited, and to modify the audit process to accommodate these views.
● Failure to continue and complete the audit cycle.
This makes it impossible to determine whether the audit has led to any improvements in
care.
● Failure to provide a supportive environment for audit.
Perceived lack of support at all stages, together with a range of structural and
organisational problems, is associated with poor progress in conducting audit. Research
has pointed to a theory–practice gap for clinicians carrying out audit, one solution being
to change the organisational culture to one in which clinical audit is supported and actively
encouraged.
● Lack of resources, especially time.
This includes lack of protected time to investigate the audit topic, collect and analyse data,
and the time to complete an audit cycle. It follows that audit should be recognised as an
important part of clinical practice and those directly involved in audit need to be allocated
protected time.
● Lack of training in audit methodology and evidence-based skills.
Health professionals and audit support staff require adequate knowledge and skills for
undertaking audit, and they should be keen to learn. Barriers identified in the literature
include a lack of training in evidence-based audit skills and the failure to apply what has
already been established.
● Cost.
It must be recognised that audit requires appropriate funding and that improvements in
care resulting from clinical audit can increase costs.

Clinical Governance Advice No. 5 8 of 10

 APPENDIX

Developing questionnaires
There is a large amount of literature on how to develop questionnaires.15,16 Some of the general
principles involved are presented here.

Questionnaires are often used as a tool for data collection. Questions may be open or closed.
Generally, questionnaire design using open questions; e.g., “What was the indication for caesarean
section?” (followed by space for free text response) is easier. However, analysis of these data is
difficult, as there will be a range of responses and interpretation can be problematic. Open
questions may be more difficult and time consuming to answer and can lead to non-reponse,
which results in loss of data.

Questionnaires can be composed entirely of closed questions (i.e. with all possible answers
predetermined). More time is needed to develop this type of questionnaire but the analysis is
generally easier. An example of this type of questionnaire is:

Which of the following statements most accurately describes the urgency of this caesarean section?
A. Immediate threat to the life of the fetus and the mother.
B. Maternal or fetal compromise that is not immediately life threatening.
C. No maternal or fetal compromise but needs early delivery.
D. Delivery timed to suit the woman and staff.

Closed questions assume that all possible answers to the question are known but not the
distribution of responses. Time and consideration needs to be given to the options available for
response as, if a desired response is not available, the question may just be missed out and it may
put people off completing the rest of the questionnaire. For some questions, the ‘other’ category
can be used with the option ‘please specify’, which gives an opportunity for the respondent to
write in a response. However, if this is used, thought must be given a priori as to how these free-
text responses will be coded and analysed. In some situations, not having a category of ‘other’ may
lead to the question not being answered at all, which means that data will be lost.

If questionnaires are developed for a specific project, they need to be piloted and refined to ensure
their validity and reliability before use as a tool for data collection. While those who developed
the questionnaire understand the questions being asked, the aim of piloting is to check that those
who have to fill in the questionnaire are able to understand and respond with ease. Questionnaires
that are not user friendly are associated with lower response rates, the quality of data collected
will be poor and hence results will be of little value.

References
1. Smith R. Audit and research. [see comments]. BMJ 1992;305:905–6.
2. Department of Health. A First Class Service; Quality in the New NHS. London: Department
of Health; 1998.
3. Department of Health. Learning From Bristol: The Report of the Public Inquiry into
Children’s Heart Surgery at the Bristol Royal Infirmary 1984–95. London: Department of
Health; 2001.
4. Commission for Health Improvement. Commission for Health Improvement response to
Alan Milburn statement today. Media statement 18 April 2002. [www.chi.nhs.uk/eng/news/
2002/apr/07.shtml] Accessed 11 September 2003.
5. General Medical Council. Good Medical Practice. London: General Medical Council; 2001.

Clinical Governance Advice No. 5 9 of 10

6. NHS, National Institute for Clinical Excellence, Commission for Health Improvement, Royal
College of Nursing, University of Leicester. Principles for Best Practice in Clinical Audit.
Oxford: Radcliffe Medical Press; 2002. [www.nelh.nhs.uk/BestPracticeClinicalAudit.pdf]
Accessed 11 September 2003.
7. Junor EJ, Hole DJ, Gillis CR. Management of ovarian cancer: referral to a multidisciplinary
team matters. Br J Cancer 1994;70:363–70.
8. Woodman C, Baghdady A, Collins S, Clyma JA. What changes in the organisation of cancer
services will improve the outcome for women with ovarian cancer? Br J Obstet Gynaecol
1997;104:135–9.
9. Department of Health. The NHS Cancer Plan. London: Department of Health; 2000.
10. Penney GC. Audit. In: O’Brien PMS, Broughton Pipkin F, editors. Introduction to Research
Methodology for Specialists and Trainees. London: RCOG Press; 1999. p. 95–106.
11. Royal College of Obstetricians and Gynaecologists, Clinical Effectiveness Support Unit. The
National Sentinel Caesarean Section Audit Report. London: RCOG Press; 2001.
12. Department of Health. Data Protection Act 1998. Protection and Use of Patient Information.
London: Department of Health; 1998.
13. McKenzie-McHarg K, Ayres S. Data management. In: O’Brien PMS, Broughton Pipkin F,
editors. Introduction to Research Methodology for Specialists and Trainees. London: RCOG
Press; 1999. p. 140–6.
14. Brocklehurst P, Gates S. Statistics. In: O’Brien PMS, Broughton Pipkin F, editors.
Introduction to Research Methodology for Specialists and Trainees. London: RCOG Press;
1999. p. 147–60.
15. Gillham B. Developing a Questionnaire. London: Continuum; 2000.
16. McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, et al. Design and use of
questionnaires: a review of best practice applicable to surveys of health service staff and
patients. Health Technol Assess 2001;5(31):1–256.

This advice was produced on behalf of the Royal College of Obstetricians and Gynaecologists by:

Dr S Paranjothy, Research Fellow, National Collaborating Centre for Women and Children’s Health, London;
Miss J M Thomas MRCOG, Director, National Collaborating Centre for Women and Children’s Health London.

Peer reviewed by:

Professor R Baker, Clinical Governance Research and Development Unit, University of Leicester;
Mr M Bocock, Clinical Governance and Audit Manager, Birmingham Women’s Hospital;
Dr P F W Chien MRCOG, Dundee; Dr G C Penney FRCOG, Aberdeen; RCOG Consumers Forum;
Royal College of Midwives; Professor S Thornton MRCOG, Coventry.

Final version is the responsibility of the Guidelines and Audit Committee of the RCOG

Valid until October 2006

unless otherwise indicated

Clinical Governance Advice No. 5 10 of 10