AMP Training

Module Ten Part Three

Audit of predetermined objectives
Reliability
Module outline
1. Reliability assertions and audit criteria
2. Basic risk rules for reliability
3. Population for testing
4. Accept-reject sampling for tests of details
5. Means of selecting items for testing
6. Accumulation and evaluation of reliability
misstatements
 AoPO Reasonable assurance: Execution per subject matter (SM)

ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population

Adjustments

Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary

For SM per indicator For SM per target Per target Per population per assertion
 ①
Reliability assertions and
audit criteria
 Reliability assertions and audit criteria

Reliability

Assertions
Substantive/
• Validity
combined approach
• Accuracy
• Completeness
Pertaining to actual performance/target
achieved

Various working papers

depending on the target
and approach to testing
Reliability - Assurance bucket
• Approach determined per assertion for a
performance indicator
• Planned approach per indicator applies to all
related targets TARGET

• Combined vs substantive

Size 3

Tests of details
Size 2 Size 2
Substantive analytical Substantive analytical
procedures procedures
Tests of details
Tests of
Tests of controls controls

Completeness Validity Accuracy

Nature of audit procedures
Purpose
To obtain sufficient appropriate evidence at the required
level of assurance
Type
• Inspection
• Observation Nature
• Inquiry
• Confirmation
• Recalculation
• Re-performance
• Analytical procedure
Nature of audit procedures
Further
Audit
Procedures

Test of Substantive
Controls Procedures

Substantive
Test of
Analytical
Detail
Procedures
Nature of audit procedures
• Can CAATS be used?
• To what extent can CAATs be used in designing
and executing test of details:

Perform more extensive testing of electronic files

Select sample events/occurrences/items/transactions
from key electronic files

Sort events/occurrences/items/transactions with
specific characteristics

Test an entire population instead of a sample.
• ISA involvement
Direction of testing
Performance records Source documents

 Validity
Accuracy Using highest
level of
Validity and Accuracy assurance
required

Performance records Source documents

Completeness
Primary vs
secondary
Substantive analytical procedures for
AoPO
Approach with care

• Suitable?
• Able to predict outcome of target?
• Logical relationship between the recorded value
and the data to be used to predict
• Plausible and predictable?
• Less effective when testing target subject to
management discretion
 ②
Basic risk response rules
Reliability
Basic risk response rule
Reliability
• Further audit procedures in responsive to risk of MM
• More persuasive audit evidence
• Circumstances appropriate for ToC
• SP’s for all material performance indicator/measure
and targets
• SP’s for significant risk SToD if fully substantive
approach
• Special procedures
Special consideration items
Performance information closing process
Rule 1 -
Further audit procedures responsive to
assessed risk of MM
• Further audit procedures responsive to the
assessed risks of material misstatement at
the assertion level; referring to the validity,
accuracy and completeness
TARGET

Size 3
proedures
Tests of details
Size 2 Size 2
Substantive analytical Substantive analytical
Tests of details procedures

Tests of controls Tests of

controls

Completeness Validity Accuracy

Rule 2 -
More persuasive audit evidence the
higher the risk

3 Significant

2 Standard

1 Minimal
Rule 3 –
Are test of controls appropriate?

• Sub-rule: If SP alone cannot provide sufficient

appropriate audit evidence
• Sub-rule: If the auditor plans to rely on the
operating effectiveness of identified controls

Similar to AFS
Rule 4 –
Always SPs for material performance
measure/ indicator and targets

Irrespective of the assessed risks of material

misstatement substantive procedures are
performed in relation to each material
performance measure/indicator and its targets

ALL material NB
Also test ALL non-
performance indicators
material indicators and
and related targets are
related targets
tested
 Testing of non-material indicators
Always: Required procedures
relevant to performance
information closing process (rule 7)

Specific procedures if affected by

specific considerations i.e.
auditing of estimates (rule 6)

Further SPs if risk significant or

standard

No further SPs required if

minimal risk and ALL conditions
are met…..
Testing of non-material indicators …

No further SPs required if minimal

risk and ALL conditions are met

The likelihood
NO significant
NO prior year due to other
risk indicators
misstatements risk factors is
are present
LOW
Rule 5 –
Specific SP’s for significant risk

3 Significant
• Auditor’s response must include substantive
procedures specifically responsive to risk
• If substantive approach followed, must include
test of details
Rule 6 –
Specific SP’s for specific consideration
items
• When a measure/indicator and any of its targets
is affected by identified special consideration,
perform required procedures
• Specific consideration?
 Auditing of estimates
 Using the work of internal auditors
 Using the work of an auditor’s expert
 Using the work of “others”
Rule 7 –
Performance information closing process
NB
SP’s related to the collation, summary and finalisation
of information on actual target achievement
a) Agree/reconcile the reported information with
underlying records
b) Examine adjustments to the reported information
on target achievement in the process of collating,
summarising and closing-off

Lead schedule
 Roll forward rules
Interim test performed

• AFS rules apply

• Refer to Chapter 10.1
 ③

Population for testing

Population
What is a population ?
Is the entire set of data from which the
auditor will select items for testing about
which the auditor wishes to draw
conclusions
At what level?
Actual target achievement for each
performance target represents a separate
population for testing
Population

What will influence the population for testing?

Nature and characteristics:

Unit of Assertions tested Stratification

measurement
i.e. direction of i.e. sub-
i.e. single /once – testing populations or
off achievements, stratums with
R-value, number, separately
percentage / identifiable
ratio/rate characteristics
 ④

Accept – Reject
Accept-reject sampling for tests of
details
 ⑤
Means for selecting items
for testing
 Selection hierarchy
Selecting all items
•
Selecting specific items only

Using a combination of specific items and audit

sampling

Audit sampling only

SToD: Approach and
accumulation
 AoPO Reasonable assurance: Execution per subject matter (SM)

ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population

Adjustments

Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary

For SM per indicator For SM per target Per target Per population per assertion
 Selecting all items
• Target is measured as a single/ once-off
achievement SToD: Testing per
population (Tab A)
• Target = 0
• Numbers, percentages, values, etc.
• Small number of items
• Significant risk with no other means of
testing
• CAATS

• Factual misstatements
 Selecting specific items

• High value items associated with R – values

• Key items that exhibit some characteristic

related to the risk assessment

Not common for AoPO

SToD: Testing per

Factual misstatements population (Tab A)
 SToD: Testing per
Sampling population (Tab B)

FIRST: Define the purpose, population and sampling unit

THEN FOLLOW THE FOLLOWING STEPS

Step 1 - Determine the sample size

Step 2 - Select the sample items

Step 3 - Perform the planned procedures on the selected

items

Step 4 - Evaluate the sample results and conclude on

sampling risk
Indicator: FTE Jobs
Indicator: % Coastline
SToD - AoPO - Testing single or
once-off achievements
 Define the purpose, population and
sampling unit
• Define the purpose of the audit procedure

• Define the error condition

• Define the population

when audit sampling is the only means of testing?

when specific and audit sampling is used?

• The definition of the population for sampling must

Refer to the assertions being tested

Direction of testing

• Percentage unit of measure

• Stratification

• Source/source information where sample is selected is complete and

accurate
• Define the sampling unit
Step 1 - Determine the sample
size
 Step 1 - Determine the sample
size
Apo tests of details sample sizes *Note*
Overall materiality = 10%
Level of assurance required

Performance Significant (= 3) Standard (= 2) Minimal (= 1)

materiality level of level of level of
assurance assurance assurance

5.0%
(10% x 50%) 60 40 20
7.5%
(10% x 75%) 40 27 14
Step 2 – Select the sample items

Step 2 – Select the sample items

Non-statistical or Statistical selection methods

Use the random selection method

Step 3 - Perform the planned procedures on the selected items

 Integration with AFS and
Compliance audit
Performing AFS , AOPO and/ or compliance audit procedures concurrently
with the financial audit procedures on the same items

• Population from which the items will be selected – must be

the same for the relevant audit areas

• Individual items – must be the same for the relevant audit

areas

• Considerations and decisions clearly documented

• References and hyperlinks to the relevant AFS and/or

compliance SToD Accumulation working papers must be
included
 Integration with AFS and
Compliance audit
Integration can be achieved by performing financial and/or compliance
audit procedures concurrently with the financial audit procedures on the
same items already selected for financial testing purposes

SToD Testing per population WP – Guidance note 32

Step 4 - Evaluate the sample
results
Misstatements identified

Disagreement:
Difference between information on actual performance supported by the
audit evidence and information as reported by the auditee

Limitation:
Audit evidence not available or not adequate
uncertain whether or not item is misstated

• Investigate and document nature, cause and circumstances (How, Why)

• Understand the inherent characteristics of the identified misstatements

Step 4 - Evaluate the sample results

Projected misstatement = auditors best estimate of

misstatement in population
Total factual misstatements in Total factual misstatements in
the sample (excl. anomalous) the sample (excl. anomalous)
(Total number of items in the (Total R-value of the sample) x
sample) x (Total number of (R-value of the population)
items in the population)
 Conclude on sampling risk
TEMOR
• Total Estimated Misstatement Occurrence Rate
(TEMOR)
• All projected + anomalous misstatements

• Calculated for quantitative evaluation of

sampling risk only
• Not transferred for accumulation to
accumulation WP
 Conclude on sampling risk
Sample risk reduced to an acceptable level?
Allowance for sample risk

Performance Overall
materiality materiality
5% / 7.5% 10%

Scenario 1 Scenario 2 Scenario 3

 AoPO Reasonable assurance: Execution per subject matter (SM)

ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population

Adjustments

Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary

For SM per indicator For SM per target Per target Per population per assertion
 6
Accumulation and evaluation
of misstatements for
reliability
Accumulating reliability misstatements

All the misstatements identified in relation to a target are

accumulated including:
From ALL the different
For assertions –
populations (or sub- From the closing
Validity, accuracy and
populations/strata – if processes
completeness
applicable)

SToD: Approach and

NB accumulation

To determine the NET effect of the misstatement on the target.

 Correction of misstatements

Net
Communicate uncorrected
Adjustments by Audit
all misstatements
management? adjustments
misstatements – calculate the
MOR per target
 Calculate misstatement occurrence
rate
• Calculate MOR based on the total uncorrected
misstatement for the target – i.e. the net effect of all
uncorrected misstatements SToD: Approach
and accumulation
• Using the applicable formula
– Single / once-off achievement - 0% or 100%
– R-value/Numbers - Total uncorrected
misstatement/ Total misstatement
– Percentage / rate - Total uncorrected
misstatement / Reported achievement as
percentage
 Programme/objective/development priority: [Provide description]
Audited value for
Performance Actual reported the target Total
measures/indicators performance uncorrected Misstatement
and their targets in the APR misstatements occurrence rate
(i.e. “Book identified for
value”) the target *Note 1*
Material measures/indicators and their related targets
Measure/indicator #1: [Provide description]
1.1 98% 87% 11% 11.2%
[Describe target] *Note 2*

> Numerator 334 296

> Denominator 340 340
1.2 590 590 0 0%
[Describe target]
1.3 1 500 1 550 50 3.3%
[Describe target]
Measure/indicator #2: [Provide description]
2.1 Approved policy No policy All 100%
[Describe target] document document
compiled
Measure/indicator #6: [Provide description]
Non-material measures/indicators and their related targets
Measure/indicator #3: [Provide description]
3.1 1 1 0 0%
[Describe target]
3.2 17 16 1 5.9%
[Describe target]
 Evaluation of reliability misstatements
relating to material targets
MOR ≥ 10%

quantitatively material and

therefore material to the
programme/objective/development priority as a
whole
Evaluation of reliability misstatements
relating to material targets
MOR < 10%
Qualitative evaluation of the misstatement
• evaluate whether the misstatement is
material, individually or when considered
together with other misstatements
• nature, cause and circumstances

• qualitative considerations Chapter 15 para 65

 Material misstatement relating to
reliability
• All targets to a material indicator are regarded
as a material target
• If material target is materially misstated,
indicator is materially misstated
• If only 1 material indicator is materially
misstated the subject matter is materially
misstated
 AoPO Reasonable assurance: Execution per subject matter (SM)

ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population

Adjustments

Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary

For SM per indicator For SM per target Per target Per population per assertion