How to Conduct a Learning Audit

By Will Thalheimer, PhD

 Will Thalheimer, PhD

Introduction

I’ve been conducting learning audits for over a decade. Over the years, I’ve learned a
ton about what works—and what doesn’t. I’ve had a number of great successes and I’ve
made some substantial mistakes as well. I’ve audited many forms of learning, including
classroom training, elearning, and on-the-job learning. I’ve worked with many types of
organizations, including huge multinationals, small elearning shops, trade associations,
foundations, and universities. In this short report, I will share my “lessons learned” and
provide you with recommendations on how you can audit your own learning
interventions.

What is a Learning Audit?

A learning audit is a systematic review of a learning program to determine the

program’s strengths and weaknesses—with the aim to guide subsequent improvements
of that learning program and/or other learning programs. Learning audits are conducted
in a high-integrity manner to ensure validity and limit bias. Full-scale learning audits are
comprehensive and involve a substantial amount of time and effort. Learning audits can
be done more quickly as long as they maintain a high-integrity systematic process.
Learning audits can be done on any type of learning initiative, including classroom
training, elearning, mobile learning, on-the-job learning, self-initiated learning, and
academic learning.

Why do a Learning Audit?

Why do all the best writers use editors? Why does software development require such
exhaustive quality-control reviews? Why do all skyscraper projects require extensive
engineering oversight?

The answer is obvious. Whenever humans work on complex endeavors—whenever

there are opportunities for mistakes and blind spots—systematic reviews by
experienced experts are required.

Human learning is very complex—infinitely more complex than rocket science. That is
why it’s critical that we support our learning-development efforts with periodic
systematic reviews. The need is made greater by the fact that many learning programs
have substantial deficiencies in terms of learning effectiveness.

Almost every learning intervention can be substantially improved to produce stronger

comprehension, enriched motivation, more long-term remembering, better supports for
learning application, and/or an enhanced evaluation approach.

2
 Will Thalheimer, PhD

Let me use my own experience as an example. In my learning audits, I’ve looked at

classroom training programs, elearning programs, and hybrids of the two. I’ve looked at
one-on-one coaching, manager-directed learning, and learning in small groups. I’ve
looked at learning programs that were highly sophisticated and ones that were strictly
low budget. I’ve looked at high-fidelity simulations, video-based case studies, lectures,
and textbooks as well. Here’s the point: No matter what learning program I’ve
examined, all of them had strengths and weaknesses. Yet each could have been
significantly improved with some relatively simple and inexpensive design changes!

Learning programs can be deficient for a number of reasons. Here’s a short list:

1. Decision-making stakeholders request poor learning-design methods.

2. Resources are not available to produce more effective designs.
3. Project timelines do not enable more effective designs.
4. Learning designers are unaware of better learning methods.
5. Learning designers are blind to opportunities for improvement.
6. Legacy designs compel reuse of poor learning methods.
7. Poor needs assessments skew content to wrong topics.
8. Poor media choices limit motivation and learning impact.
9. Adherence to rigid instructional-design rules hurts learning.

By doing a learning audit, deficiencies will be uncovered that can be targeted for
improvement. Sometimes these improvements can be made by the learning-design
team itself. Other times the learning audit gives us the ammunition to convince
stakeholders of the possibility and importance of making learning-design improvements.

3
 Will Thalheimer, PhD

Be Careful! Which Standard Should You Use?

You wouldn’t evaluate the engineering integrity of a modern skyscraper using criteria
developed by a high-school study group, nor would you use a 1947 nurses manual to set
guidelines for today’s sophisticated nursing tasks. It’s the same with a learning audit.
The key is to start with a valid set of standards.

While it might be tempting to rely on our “common sense” in developing standards, too
often our common sense in the learning field leads us astray. Until recently, all the
following were deemed to be simple common sense in the learning field. Yet, for each of
the following, common sense was shown to be dead wrong:

1. Training should be designed to handle different learning styles. 1

2. Feedback should always be delivered immediately. 2
3. Learners can be trusted to make good learning decisions. 3
4. Smile sheet results are strongly correlated with learning results. 4
5. Massed practice is more effective than spaced practice. 5
6. eLearning is more cost effective than classroom training. 6

Fortunately, over the past several decades, learning researchers have codified an array
of factors that enable learning to be effective. These can be found in the Decisive Dozen,
in the principles set out in the Serious eLearning Manifesto, and in books like Make It
Stick: The Science of Successful Learning. While research, too, can have blind spots, it
gives us our best benchmark—as long as it is compiled with practical wisdom.

Here are just a few of the many things our learning audits ought to assess:

1. Whether the learning program supports remembering—utilizing such learning

methods as realistic practice, spacing of repetitions over time, and setting
situation-action triggers.
2. Whether the learning program propels learners to be motivated to apply what
they’ve learned.
3. Whether learners have sufficient after-learning support and resources to enable
them to be successful in applying what they’ve learned.
4. Whether the learning program provides prompting mechanisms—like job aids—
to support learners later in their performance situations.
5. Whether the learning program provides sufficient measurements to enable
course developers to get feedback and make improvements.

1
(Pashler, McDaniel, Rohrer, & Bjork, 2008).
2
(Thalheimer, 2008a, 2008b).
3
(Kirschner & van Merriënboer, 2013).
4
(Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008).
5
(Carpenter, Cepeda, Rohrer, Kang, & Pashler, 2012).
6
(Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012).

4
 Will Thalheimer, PhD

Examining Inputs and/or Outputs

Learning audits can target the inputs and/or outputs of a learning program. The inputs
are the factors baked into a learning program’s design and delivery. The outputs are the
results obtained from deploying the learning program to learners. The following chart
lists some common inputs and outputs:

Examples of Inputs Examples of Outputs

Number of learners served Learner responses on smile sheets
Timing of learning delivery Learner scores on final tests
Support for comprehension Learner performance on case studies
Support for remembering Subsequent learner job performance
Support for motivation Subsequent team performance
Support for after-learning application Subsequent business results

The most comprehensive learning audits will examine both inputs and outputs. For
inputs, a full learning audit will research-benchmark the program, comparing its
instructional design approaches to research-based best practices. In addition to auditing
fully-developed programs, we can also audit designs and prototypes.

For outputs, the most comprehensive learning audits will examine (1) comprehension
gains due to the learning program, (2) decision-making competence, (3) level of
remembering, (4) amount of actual performance improvement, (5) the strengths and
weaknesses of the learning-measurement system, and (6) the learning program’s fit
with the organization’s business models.

Of course, such fully comprehensive audits are costly and time consuming. More
importantly, depending on your goals for your audit, it is very unlikely that you will need
to be so comprehensive. Indeed, I have never been asked to conduct a learning audit
comprised of all the assessments that might be done.

The advantage to research benchmarking on the input side of the ledger is that the
focus is on design elements that are modifiable and which can be targeted for
improvement. The advantage to doing an evaluation study on the outputs—if these
evaluations are well done—is that we are able to examine the learning program’s actual
results. Ideally, an audit that examines both inputs and outputs will connect the dots
between the output results and the input design elements. For example, the audit might
point out that the after-training decline in decision-making competence (as measured at
the one-month mark) is probably due to the learning program’s lack of support for
remembering—or, more specifically, that the program didn’t provide enough realistic
decision-making practice spaced over time. Obviously, there are many other ways to
connect the dots.

5
 Will Thalheimer, PhD

Using an Effective Learning-Audit Process

Learning audits will be more successful if they are conducted using a proven process.
Some variant of the following process is likely to enable success:

1. Solidify Sponsorship
Find a sufficiently powerful decision maker to sponsor the effort. Without a
sufficiently powerful person—a person who can make decisions, rally political
support, provide time and resources—no change effort is likely to be successful.
Learning audits are change efforts, or they are not worth the investment!

2. Enlist Stakeholders
Enlist a sufficient number of key stakeholders to agree to or accept the audit
process. While it is desirable to get the full support of these key stakeholders—
instead of just their acquiescence—a well-conducted audit review meeting can
subsequently gain their support.

3. Use a Two-Phase Process

Create a two-phase process, with an initial exploratory phase dedicated to
learning enough about the learning program to plan and propose the second
audit phase. While a two-phase approach may not be necessary for audits of
very short learning interventions (for example, a 45-minute elearning program),
it is generally worthwhile to do enough exploratory work to set expectations
about the focus of the audit.

4. Budget the Exploratory Phase

Agree to budget and time constraints for the exploratory phase of the audit.
This is an obvious step when using an auditor from outside your organization,
but setting a clear expectations can be just as useful when an internal person or
team is conducting the audit—as it ensures everyone is on the same page.

5. Conduct Exploratory Phase

Conduct the initial exploratory phase of the audit, doing a quick-and-dirty
review of the learning program and interviewing a small subset of key
stakeholders—all for the purpose of designing the audit phase. Produce a short
report, using presentation software (like PowerPoint or Keynote) and share
recommendations for the second phase—the audit phase. If possible, avoid the
temptation to report any conclusions you might be forming. If compelled to
report on your initial conclusions, frame them as “hypotheses” to be more fully
tested during the audit phase.

6. Budget the Audit Phase

Agree to budget and time constraints for audit phase.

6
 Will Thalheimer, PhD

7. Conduct Audit Phase

Conduct audit activities, allowing for slight modifications to the original plan as
new information comes to light.

8. Prepare Stakeholder Level-Setting Events

Prepare one or more mini-workshops to educate your stakeholders on the
rationale and reasoning behind the audit metrics you are using. For example,
when I use the Decisive Dozen to audit a learning program, I typically provide a
three-hour introduction to the research, presented in a manner to surface
common misconceptions and bring to light fundamental learning-design
principles. When I audit a program’s learning-measurement approach, I will also
include a primer on learning measurement.

While these level-setting events may seem superfluous, they are absolutely
critical to ensure that everyone is on the same page. Although they do take
extra time, they enable much more productive discussions when the audit
findings and recommendations are being discussed, and when solution
brainstorming is being conducted.

Many years ago, I tried to share audit findings without first providing these
educational sessions. The results were disastrous! My clients couldn’t
understand where my recommendations were coming from. They disagreed
with each other about how learning worked. Our discussions got hijacked by
concerns over minimally-important learning factors when we should have been
focusing on the most important learning factors. In short, my early audits failed
to communicate with clarity. They created situations where my audit
participants were all over the place in their conclusions, making solution
brainstorming and prioritization a most difficult endeavor.

These educational sessions are critical for three primary reasons. First, the
research-based standards you use may not be fully understood by all your
stakeholders. As we’ve already seen, some of the recent learning research is not
intuitive. Also, you may have stakeholders who are not immersed in learning as
their profession. Second, because the learning field does not always have a clear
and consistent body of knowledge, practitioners in the field often approach
learning from widely disparate perspectives. By providing educational sessions,
you can heighten attention toward the standards you are using. Finally, some
learning factors are more important than others. It helps to get everyone
focused on factors that are of the highest importance rather than factors that
are less important.

7
 Will Thalheimer, PhD

I hope I’m making a strong case here for the importance of these educational
sessions. Let me add one final thing to help persuade you. These educational
sessions are so important that I will NOT do a learning audit if a client doesn’t
agree to include them!

Now that you’re persuaded, there are a few other things you absolutely must
know about how to prepare for these educational events. First, you have to give
people more than just recipes or rules. You have to provide them with a deep
understanding of human learning. You want them to be able to think for
themselves about learning—to flexibly and creatively apply learning knowledge
in making instructional-design decisions. Second, to support the first goal,
you’ve got to give them time to think through the factors you present—
preferably by having them work through realistic instruction-design decisions—
reflect on their own learning approaches and learn from a breadth of different
perspectives.

9. Prepare Audit Report

Produce the audit report you will present to your stakeholders. Again, I
recommend that you create this report using PowerPoint or Keynote, because,
when you present your findings, you want everyone to pay attention to the
same thing at the same time.

Your audit report should consist of the following sections:

A. An Introduction. Your introduction should consist of a schedule of the

discussions that are planned, a list of the promised deliverables, an
explanation of the audit process, an acknowledgement that the findings
can bring both joy and disappointment, an acknowledgement that the
auditor is likely to have blind spots, and a call to make this as much of a
collaborative process as possible.

B. Findings Indexed to Your Standards. For each of your standards, you

ought to provide a short list of your overall findings.

C. Exemplars of Specific Findings. It’s critical to show specific instances of

good and bad instructional-design elements. The more concrete you can
make these, the better. For example, screenshots of elearning programs
provide a great context to show stakeholders what is strong or what is
weak. In classroom programs, exemplars can include slides, workbook
pages, or detailed descriptions of classroom events. How many
instances should you show? This depends on the learning intervention.
To provide some examples, in a recent learning audit of a 90-minute
elearning program—one where I was asked to do just a basic, surface

8
 Will Thalheimer, PhD

review—I shared 50 screenshots and added my comments to each. For

a massive elearning program, where the learners were expected to
spend hundreds of hours in learning, I provided over 350 screenshots
with commentary.

D. Overarching Findings and Recommendations. In this section the auditor

(or auditors) puts forth his or her assessment of the overarching themes
of the audit findings and a list of specific recommendations. In addition,
the auditor may create a recommendation of priorities for
improvement, a rough schedule of suggested improvements, and/or a
tentative strategic plan based on the audit findings. While this part of
the report will begin as a one-way communication from the auditor to
the stakeholders, it should be conducted with an openness to seek input
and to hear and respond to critiques from the stakeholders.

E. Brainstorming Opportunity. Your audit report ought to include some

sort of marker that lets all the participants know there will be an
opportunity for them to brainstorm solutions and plan for
improvement. Moreover, as you prepare for the report, you should be
planning on how to facilitate this brainstorming session.

10. Deliver Educational Event (or Events)

Deliver the educational session to all key stakeholders shortly before revealing
the audit findings. These educational level-setting events must be delivered
immediately prior to the audit findings to ensure they have the maximum effect
in helping everyone get on the same page.

11. Reveal Audit Findings

Reveal audit findings to all key stakeholders. While it is not absolutely
necessary, it is extremely helpful to do this in a face-to-face meeting, so body
language and tone of voice can facilitate a beneficial feeling of trust.

12. Share Recommendations

Put forth recommendations in a manner that enables stakeholders to discuss
confusions, understand the ramifications clearly, question your conclusions,
piggyback on ideas, reflect on the findings, and generally act as a contributing
member of the team. On the other hand, do what you can to have participants
hold their “solution ideas” until the next item on the agenda—the brainstorming
session. This can often be accomplished by handing out medium to large sticky
pads and having participants “parking lot” their solution ideas.

9
 Will Thalheimer, PhD

13. Brainstorm Solutions

Brainstorm solutions with key stakeholders. Here, the auditor or a neutral third
party should facilitate.

A. Setting Out a Brainstorming Plan

For brainstorming to be effective, it has to go well beyond getting
people into a room and generating ideas. It’s helpful to develop a more
sophisticated plan for your brainstorming session. It’s also helpful to let
your participants know what that plan is as you begin.

B. Individual Brainstorming
Begin by having each participant individually write down a list of
solution ideas. Alternatively, you (as the auditor) can prepopulate
specific solution ideas from your list of recommendations and ask the
participants to add to the list. Also, you can offer a set of categories and
ask the participants to generate solution ideas within each of those
categories. The key here is that you may need to support the
participants in thinking through all the possibilities that might be
available. You want to avoid having them go down one or two paths and
forget other important solution opportunities. On the other hand, you
don’t want to be so structured that your participants aren’t asked to
make substantive contributions.

C. Group Brainstorming
Only after individual brainstorming should you initiate group
brainstorming. Research shows that individual brainstorming generates
more creative ideas and ideas along more varied dimensions—so it’s
critical to do individual brainstorming first.

D. Categorizing
Because you’re going to get redundant ideas and ideas that overlap, it’s
helpful to get the participants to group the ideas into conceptual
categories. This can easily be accomplished by placing all the ideas on
individual stickies, placing all the stickies on a wall, and having
participants as a group move the stickies around to form clusters of
ideas. It can be helpful toward the end—when the clusters have been
more-or-less finalized—to add labels and connecting lines to the various
clusters to help make sense of them.

E. Reflection
It’s really helpful at this point to have the group sit down again and
spend some time reflecting on their initial clustering scheme.

10
 Will Thalheimer, PhD

14. Prioritize Solutions and Develop Timeline

Prioritize solutions and develop a rough timeline for implementation. It’s often
helpful here to create a grid of sorts—or at least think about prioritizing—based
on two dimensions.

A. Prioritize items on a scale from easy to difficult in terms of

implementation.

B. Prioritize items on a scale from least important to most important in

terms of benefits.

15. Enlist People in the Implementation Improvement Plan

Enlist decision-makers to support the Implementation Improvement Plan. Enlist
key stakeholders in championing and/or carrying out the tasks involved the
implementation improvement plan.

16. Budget the Implementation Improvement Plan

Agree to budget and time constraints for the implementation improvement
plan.

17. Monitor Progress on the Implementation Improvement Plan

Set up milestones and follow-through with oversight to monitor the progress on
implementing the audit-generated improvement ideas.

18. Make Midcourse Corrections

Based on the findings of your ongoing monitoring, make midcourse corrections
that enable you to get the most value for the costs and exertions being incurred.

19. Conduct Follow-up Learning Audits

Periodically, as needed, conduct additional learning audits to monitor your
progress on previously-audited learning programs and to expand the audit
process to previously-unaudited learning programs.

11
 Will Thalheimer, PhD

Audit Data-Gathering Options

There are many ways to gather data on learning audits. Here is a brief list to give you a
sense of the variety of methods available to you.

• Interview learners, learners’ supervisors, learning designers, learning

developers, learning deliverers, and other organizational stakeholders.
• Focus-group some of these folks—most likely in groups of similar individuals.
• Survey these folks or a subset of these folks.
• Job-shadow people as they learn and work on the job.
• Research-benchmark the learning program or prototype (or design intentions)
based on a validated list of key learning factors (such as the Decisive Dozen).
• Analyze organizational artifacts (like company newsletters and bulletin boards)
and other communication devices from a learning perspective.
• Create a list of the learning media that have been utilized.
• Create a list of the available learning media.
• Analyze the learning measurement approaches utilized.
• Review smile sheet results.
• Review results of learning assessments—especially assessments that go beyond
memorization questions to scenario-based decision-making, case studies,
simulations, and realistic hands-on exercises.
• In addition to reviewing results that are assessed during or immediately after
learning, seek to review results that assess learning after a delay of a week or
more.
• Review on-the-job performance results that are routinely captured.
• Review business results, especially those that are linked to learning.
• Review the quality and use of prompting mechanisms (like job aids).
• Review the quality and use of on-the-job learning affordances, including
coaching, social media, knowledge-management systems, team learning, etc.
• Review the supports in place for creativity-based insight learning.
• Review the supports in place for after-learning application.
• Develop and deploy improved smile sheets, learning assessments, and
performance assessments.
• Conduct A-B testing on different versions of the same learning program.

This list could go on and on, but that makes it more difficult—not less. The key is
deciding where the most value comes from. This will depend partly on the goals you
have for your learning, your constraints, and a host of other factors—but there are
some generalizable truths here.

12
 Will Thalheimer, PhD

Research Benchmarking
The most important thing you can do—in terms of data gathering—is to research-
benchmark your learning program. Research benchmarking is at the top end of potency
because (1) it targets learning-design issues that really make a difference in learning
effectiveness; (2) it targets learning-design issues that are under the control of us as
learning professionals; and (3) it is based on research-validated learning factors.

Learner Verbal Feedback

Another important data gathering option includes getting learner feedback, especially
learner feedback as recorded in their actual statements about the learning. While
learners don’t always know the best way for them to learn, their statements resonate
deeply with decision makers.

Interviewer: “Is there anything you wish you could do

differently in developing your people?”

Manager: “First thing I would do is abolish about 80-90%

of the elearning they have to take.”

One statement like this may not register, but, after about five like this, organizational
decision-makers sit up and pay attention.

Of course, given the subjective nature of these responses, they must be augmented with
other data and viewed based on research-based perspectives on learning.

Evaluating Learning Measurement Practices

Because our learning measurement practices give us our best source of ongoing
feedback about the success of our learning initiatives—and because so many
organizations completely fail in getting good feedback—it’s essential to get a sense of
how well learning measurement is being used to get valuable and unbiased feedback.

Utilizing Scenario-Based Decision Making

While it is desirable to measure performance in real-world contexts or in high-fidelity
simulations, it is often too expensive and logistically infeasible. The next best thing is
measuring scenario-based decision making.

One of the key goals in learning is to help learners associate situational cues with
desired actions.

Situation – Action

Here are some examples. We want our supervisor learners to know which situations
require them to seek input from their direct reports. We want our sales-person learners

13
 Will Thalheimer, PhD

to know which conversational response to make to each potential objection they hear
from their customers. We want our statistics students to know which statistic to use
given the research question at hand. If our learning designs don’t provide learners
practice in situation-action pairs, then our learning is not supporting performance.

If we’re auditing learning, one of the strongest ways to determine whether the learning
is providing sufficient practice is to measure our learners’ ability to make scenario-based
decisions. In our audit work, we can present learners with various situations and ask
them what actions should be taken. The key, of course, is that our scenario-based
questions have to be well conceived to provide the most critical realistic cues, and they
have to be set in backgrounds that are appropriate to learners’ future performance
situations. How to design well-crafted scenario-based questions is beyond the scope of
this report—but, suffice it to say that, while they are easier to develop than full-blown
simulations, designing scenario-based questions still requires substantial know-how.

Data Triangulation
Because all data gathering is probabilistic—that is, it is only approximating the reality on
the ground—it is best to use more than one source of data in a learning audit. While
research benchmarking can stand alone as an audit method, none of the other data
gathering approaches should be utilized on their own. Even research benchmarking
should be augmented when time and resources allow.

Having multiple data sources not only enables you to corroborate your findings; it also
helps you make a stronger case to your stakeholders.

14
 Will Thalheimer, PhD

Examples of Learning Audits

Large Retail Drugstore Chain

The Director of Learning of one of the largest U.S. drugstore chains wanted to see how
well her learning function was supporting on-the-job performance of employees in the
retail stores—and she wanted to know what her learning group might be able to do to
improve their results. Also, the company’s elearning courses were not as well received
by learners as their previous classroom courses—and she wanted to know what might
be done.

Given that the goal was performance improvement as supported by the whole learning
function, this learning audit looked not only at elearning programs, but also at the
whole learning ecosystem in the retail stores. eLearning programs were audited based
on a rubric of learning-research considerations. A classroom-based course for assistant
managers was observed over several days. In addition, structured interviews and focus
groups of different learner groups were conducted. In addition to learners, other key
stakeholder groups were interviewed, including senior leaders, key managers, and
learning professionals. Employees were observed in their jobs and some were job
shadowed. In addition, company communications were evaluated as learning tools,
including such artifacts as the company magazine, bulletin boards, and the company’s
website portal.

Among many findings, one of the most notable was that employees reported that they
learned from those people who they worked with the most. This finding had significant
implications. First, it showed that the way people interact with each other can
significantly impact learning—and that more attention might be paid to helping people
learn from each other. Second, where the organization had been sending itinerant
trainers around to the stores, such a strategy was not working as well as it might
because of the relatively few touch points between the trainers and employees.

The audit also found that on-the-job coaching was not always effective. Employees
tended to overuse “telling” as a coaching technique and failed to engage their coachees
in practice. As the research makes clear, real-world practice and feedback are especially
beneficial in supporting comprehension and remembering.

The audit also gave the organization a wake-up call on their elearning. Where previously
they had accepted low elearning ratings as routine, getting the research benchmarking
results and witnessing actual comments from disgruntled learners made the learning
team realize that significant changes were needed. Indeed, when the head of elearning
assisted in the focus groups, she was completely taken aback by the harsh criticism the
elearning received. To her credit—and the organization’s—this feedback propelled the
team to rethink its elearning designs.

15
 Will Thalheimer, PhD

Educational Foundation’s Blended eLearning Materials

An educational foundation was making its first foray in using elearning to help educate
one of its critical audiences: college students who were learning to start their own
businesses. The Foundation typically created classroom learning materials and handed
them over to faculty members who taught courses with the materials. Faculty members
typically rated these materials very highly. Their new effort combined classroom
materials with elearning materials—but they were still provided to faculty members to
utilize within their own courses.

The Foundation’s previous practice was to get feedback from faculty members on the
program. They often did this by inviting faculty members to the Foundation’s offices for
meetings and by keeping in close contact with faculty members who were teaching with
their materials. They also surveyed faculty members periodically.

The learning audit was aimed at getting an overall evaluation of the new program. To
keep it consistent with the Foundation’s past learning-measurement practices, faculty
members using the new elearning/classroom hybrid course were surveyed using the
same questions asked of those who taught with the classroom-only materials. Faculty
members were also interviewed.

In addition to seeking faculty input, many other data gathering efforts were made. The
elearning and classroom portions of the course were research-benchmarked. Scenario-
based questions on the learning program’s key concepts were validated with about ten
subject-matter experts and then deployed using cloned items on pretests, immediate
after-learning tests, and delayed after-learning tests. Finally, student business plans
were evaluated by expert business-plan reviewers.

Among many findings, one of the most notable was that, while faculty members rated
the course materials very highly—consistent with the Foundation’s previous evaluations
of their classroom programs—learners rated the learning as mediocre. Even more
problematic was that learners did not improve from pretest to posttest and their
business plans were poorly rated.

The research benchmarking review found that the program did not sufficiently utilize
learning methods focused on realistic decision-making, tending to focus on providing
learners with a large amount of knowledge dissociated from business decision making.

16
 Will Thalheimer, PhD

Next-Generation MOOC (Massively Open Online Course)

A trade organization had hired an outside vendor to create a MOOC to teach one of the
courses it offered to its membership—a membership that was required to take
continuing education courses to maintain their professional credentials. The learning
audit was intended to be a quick-and-dirty audit using research benchmarking alone.
The MOOC was still in development and was being used in a pilot test.

The learning auditor engaged the MOOC as a learner might, and augmented this by
doing activities more than once, attempting both usual and unusual actions—capturing
screenshots and providing commentary on each. A full audit process was not utilized.
Instead, the auditor was tasked with reviewing the MOOC based on his background
knowledge—which included a background in both learning research and elearning. After
generating an initial review, the auditor utilized the Decisive Dozen to double-check
whether he had missed any concerns. After adding some additional insights to his
review, the audit review—an annotated PowerPoint deck with screenshots and
commentary—was provided to the stakeholders for their review.

The audit revealed several things that could be improved based on the pilot. First, there
were some minor issues inherent in any pilot program: navigation that didn’t work quite
right, feedback that was misaligned to the wrong response, confusing content, etc.
Second, there were a few missing instructional-design elements that weren’t utilized
well enough, including the use of job aids, additional practice, and encouragement to try
this back on the job.

On the other hand, the program was led by a brilliant, highly-credible instructor—and,
most importantly, someone who understood the situation-action contingencies within
her area of expertise. She nicely used her situation-action knowledge to highlight best
practices and common mistakes to avoid. This “next-generation” MOOC went beyond
most MOOCs by presenting a very rich video-based case—with professional production
and acting. The case asked learners to make decisions, providing them with realistic
practice. The MOOC also engaged learners with well-designed social-media elements:
for example, responding with specific insights about decisions in the case.

17
 Will Thalheimer, PhD

Classroom Training
A large government agency was concerned that its subject-matter experts were doing
an inadequate job training their field professionals. An outside auditor was called in to
analyze one of its courses. The auditor sat in the four-day course as a learner, examined
the learning materials, interviewed instructors, and spoke with students during breaks
and after the course.

The course was research-benchmarked against the Decisive Dozen. It was also
benchmarked against notions found in the Situation-Based Instructional-Design
Approach. Finally, the course was evaluated based on the Full-Source Learning-
Evaluation approach, which focuses on the learning measurement methods used to
provide feedback for course improvement. A written report was developed based on
the audit findings.

Among many other findings, the audit found that the course instructors were highly
credible and well organized. Unfortunately, too much material was presented and not
enough realistic practice was included in the course. While the instructors attempted to
increase “interactivity” through periodic quizzes focused on knowledge from the course,
these quizzes did not focus on real-world decision-making. The quizzes did have their
intended effect: they encouraged learners to pay attention to individual knowledge
morsels, with some learners even studying their notes overnight to do well on the
quizzes, which had been set up as a competition between teams. Unfortunately, while
this did reinforce knowledge acquisition, it hurt learners in thinking about overarching
themes and situational decision making.

The analysis of the learning-measurement approaches showed glaring weaknesses, with

poorly designed smile sheets producing feedback suggesting the course was
successful—when the research benchmarking showed numerous trouble-spots in the
course’s instructional design.

Reflections on the Four Example Audits

The four example audits shared above show that audits can be used for many types of
organizations and for many different types of learning interventions. They also show
that many varieties of data gathering approaches can be utilized successfully. Finally,
they demonstrate that every learning audit will find strengths and weaknesses.

18
 Will Thalheimer, PhD

Do it Yourself or Bring in an Outsider?

Every organization that conducts training or educational programs should periodically

audit its learning programs—at least a small subset of them every year—to ensure that
learning is creating maximum benefits.

While it is often helpful to bring in outside auditors—to provide an unbiased outside

perspective—many organizations, especially large ones, should develop a learning-audit
capability in-house. An in-house capability enables more frequent reviews and ensures
that workplace learning professionals on staff are developing and maintaining research-
based knowledge about learning. By having this knowledge in-house, research-based
ideas are likely to percolate more freely throughout the organization—enabling
learning-design thinking that will nurture improvements in learning interventions.

To get your organization up to speed on how to conduct learning audits, the following
process will be helpful.

1. Gather your key learning professionals together to discuss the audit idea.
2. Have them read this document and discuss thoroughly, raising both benefits
and obstacles inherent in learning auditing.
3. Get general agreement among your key stakeholders to explore the audit idea
further.
4. Have one or more of your organization’s learning programs audited by a trusted
outside learning auditor.
5. Discuss and decide whether to create an in-house learning-audit capability.
6. Select your initial team of in-house auditors.
7. Get your team trained in auditing, and particularly in research benchmarking.
8. Select one or two courses to pilot your team’s new skills. Utilize a trusted
outside learning auditor to coach your team through their first audit. Get
feedback and develop lessons learned.
9. Ramp up your new team’s capability and develop an auditing schedule for your
organization’s learning programs.
10. Periodically—perhaps every other year—have your in-house team audit a
learning program in parallel with an outside learning auditor. This will ensure
your team is continuing to produce the highest-quality audits and it will give any
new members of your audit team feedback from an expert auditor. Consider
using multiple outside learning auditors to get varied perspectives.

19
 Will Thalheimer, PhD

There are, of course, advantages and disadvantages to bringing in an outside auditor

and to handling audit responsibilities in-house. The following table highlights some of
these advantages and disadvantages.

In-House Auditors Outside Auditors

More available Experienced
More likely to see constraints Unbiased
Develops in-house competencies Knowledgeable of research
Advantages Motivates internal team Has seen more learning exemplars
Less external costs Brings credibility
Able to do more audits Brings outside set of eyes

Less time to build learning More costly

Not as knowledgeable We must hire well
Disadvantages Not as credible Takes time to learn organization
Worry about bias Fewer audits are possible

20
 Will Thalheimer, PhD

Final Thoughts

We humans are imperfect. Even when we do our best, we introduce flaws into our
work. When our work is complex, we need others to help us see what we can’t see.

This document was edited by a professional copyeditor, because it’s good to get an
expert review of one’s work.

Are your organization’s learning programs worth a second look?

Work-Learning Research, Inc.

For over a decade and a half, Work-Learning Research has been creating almost-
perfect research-to-practice reports and sharing research-based wisdom through
keynotes, conference presentations, workshops, blog posts, articles, and more—very
often offering free research-based information to the learning field.

You can make it possible for us to continue our work by hiring us and by letting others
know about the great work we do.

Check out our website dedicated to learning audits: LearningAudit.com

How to be in touch:
Phone: 888-579-9814 (United States)
Email: info@work-learning.com
Sign Up for News: http://www.work-learning.com/sign-up.html
Main Blog: http://willatworklearning.com
Twitter: @WillWorkLearn
LinkedIn: http://www.linkedin.com/in/willthalheimer

Websites:
Work-Learning Research
LearningAudit.com
SubscriptionLearning.com
AudienceResponseLearning.org
Willsbook.net
SmileSheets.com (in Development)

21
 Will Thalheimer, PhD

Will Thalheimer, PhD

Will Thalheimer is a learning expert, researcher, instructional designer, business

strategist, speaker, and writer. Dr. Thalheimer has worked in the learning-and-
performance field since 1985.

He was the project manager for the first commercially-viable computer-based

leadership simulation, The Complete Manager. He led the Strategic Management
Group’s product line, Leading for Business Results, increasing revenues fourfold. He
has trained managers to be leaders at numerous Fortune 500 companies, teaching
such topics as leadership, persuasion, conflict resolution, and business strategy. He
has led change management efforts and workshops.

In 1998, Dr. Thalheimer founded Work-Learning Research to bridge the gap between
research and practice, to compile research on learning, and to disseminate research
findings to help chief learning officers, instructional designers, trainers, elearning
developers, performance consultants, and learning executives build more effective
learning-and-performance interventions and environments. He is one of the authors
of the Serious eLearning Manifesto.

His clients have included giant multinationals, elearning companies, government

agencies, and institutions of higher learning. Short list: Walgreens, UNUM,
Microsoft, MIT, Pfizer, Allen Interactions, Type A Learning Agency, eInstruction,
Monitor Group, ADP, Questionmark, Midi Compliance Solutions, Facility Einstein,
Defense Intelligence Agency, The eLearning Guild, Rockwell, Raytheon, Boeing,
Kodak, AGFA, AMD, PPG, Nabisco, Ericsson, Abbott, Novartis, SMG, and the U.S.
Postal Service. His research and writings have led the field in providing practical
research-based recommendations through his online publications), published
articles, his industry-leading blog (www.willatworklearning.com) and Twitter
(@WillWorkLearn).

Dr. Thalheimer speaks regularly at national and international conferences. His

conference presentations always receive numerous evaluation-sheet comments like
the following: “This was one of the best presentations I attended—solid information
delivered in a style that helped me learn.”

Will holds a BA from the Pennsylvania State University, an MBA from Drexel
University, and a PhD in Educational Psychology: Human Learning and Cognition
from Columbia University.

22
 Will Thalheimer, PhD

Research

Alliger, G. M., Tannenbaum, S. I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A
meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-
358.

Brown, P. C., Roediger, H. L. III, & McDaniel, M. A. (2014). Make it stick: The science of
successful learning. Cambridge, MA: Belknap Press of Harvard University Press.

Carpenter, S. K., Cepeda, N. J., Rohrer, D., Kang, S. H. K., & Pashler, H. (2012). Using
spacing to enhance diverse forms of learning: Review of recent research and
implications for instruction. Educational Psychology Review, 24(3), 369-378.

Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban
legends in education. Educational Psychologist, 48(3), 169-183.

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts
and evidence. Psychological Science in the Public Interest, 9, 105-119.

Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of
training and development in organizations: What matters in practice. Psychological
Science in the Public Interest, 13(2), 74-101.

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review
and meta-analysis of the nomological network of trainee reactions. Journal of Applied
Psychology, 93, 280-295.

Thalheimer, W. (2008). Providing Learners with Feedback—Part 1: Research-based

recommendations for training, education, and e-learning. Available at:
http://www.work-learning.com/catalog.html.

Thalheimer, W. (2008). Providing Learners with Feedback—Part 2: Peer-reviewed

research compiled for training, education, and e-learning. Available at:
http://www.work-learning.com/catalog.html.

23