The Journal of Engineering Education

July - 2013

2. RUBRICS: AN EFFECTIVE ASSESSMENT TOOL

FOR OUTCOME-BASED ACCREDITATION
Prof. Datta Dandge*, Prof. Dr. Suhasini Desai**

(Reprinted from EEFs Journal of Engineering Education, Q. I. No. 103, Jan. 2013.)

Abstract
New guidelines released by NBA signal a paradigm shift in engineering educational system. The
guidelines follow Washington Accord of which India is a provisional member. With this shift, the focus
of programme evaluation by NBA panel has changed from outputs to outcomes. In fact, outcomes
attainment measurement methods and their review process form an important set of documentation for
new NBA. NBA guidelines also make a mention of rubrics designed to measure outcome attainment by
students. The present paper discusses the evolution of accreditation process from minimalist model to
outcomes-based model. It discusses the Graduate Attributes which are common across programmes and
the concept of rubrics for outcomes measurement.

note here that all members of Washington

Accord have Outcome-based Accreditation and
they have accepted a defined set of Graduate
Attributes (GA), to be demonstrated by their
graduates.

Washington Accord and NBA:

The Washington Accord is one of the
multinational accords that recognize accredited
programmes across countries. It recognises the
substantial equivalency of the programme
accredited by those bodies, and recommends
that graduates of the programmes accredited
by any of the signatory bodies be recognised
by the other bodies as having met the academic
requirements for entry to the practice of
engineering. The National Board of Accreditation
became a provisional member of the Washington
Accord (WA) in 2007. If India is to reap the
benefits of its large human resource and become
global supplier of trained HR, it has to aim at full
signatory member status of WA. With this
objective, a robust accreditation system is being
implemented by the NBA. The present NBA
accreditation requirements are almost matching
with ABET guidelines and are focussed on
outcomes-based education. It is important to

How NBA has evolved: From processes outputs to outcomes

Accreditation model has evolved over time
from Prescriptive to Outcome-based.
Prescriptive model is a minimalist model that
provides prescription for minimal core and
specifies very general parameters like student/
faculty ratios, courses, internships, checklists,
etc. It does not encourage continuous
improvement. Input-output model prescribes the
inputs in terms of infrastructure, financial
resources, laboratory equipment, curriculum and
faculty composition and output is measured in
terms of number of students graduating,
examination performance, success in job

*, ** Maharashtra Institute of Technology, Pune.

14

The Journal of Engineering Education

July - 2013

summarized in Table 1 below.

market or other relevant indicators. It encourages

continuous improvement in quality of outputs.
But, since outputs are defined at programme
level, they do not reflect the actual competencies
acquired by an individual student during his
graduation from the programme. Outcomesbased model that is being advocated presently
is not only an improvement over the earlier
models but marks a paradigm shift in engineering
education. It focuses on objectives and
outcomes of the program demanding explicitly
stated philosophies and principles of education
program and seeks to evaluate how well these
objectives are being met. In short, while old
output assessment model served as an
indirect measure of a programs level of
effectiveness, the present outcomes
assessment-based accreditation insists on
having direct measures in terms of program
outcomes, of what the programme has done
with its capacity and capability to contribute
to the growth in individual student learning.

Programme Outcomes (POs) describe

what students are expected to know and be able
to do upon graduation. These relate to the skills,
knowledge, and behaviour that students acquire
as they graduate through the programme.
Evidence of measurement and attainment of
these outcomes form an important part of the
documentation to be provided during
accreditation process. Reliable assessment of
outcomes also provides crucial data required in
decision making so as to monitor programmes
5
Use Results
for Decision
Making

1
Establish
Vision and
Mission

2
Formulate
objectives and
Outcomes

4
Evaluate
Findings from
Assessment
Process

Three stages of accreditation evolution are

3
Design and
Conduct
Assessments

Fig. 1: Continuous Process Improvement Cycle

Factors evaluated in accreditation

Inputs

(1) Minimalist Process

Model

(2) Outputs-based Model

(3) Outcomes
based
Model

Student Background

Courses offered
Services
Infrastructure

Results
Contact hours
Faculty retension rates
Placement statistics

Students Knowledge
Student Skills
Student Behaviours

Faculty Background

Teaching loads
Class sizes

Research Grants
Publications
Professional development

Teaching effectiveness
Research impact on
outcome attainment

Educational Resources

Policies
Procedures
Governance

Statistics on Resource
Availability and
Participation Rates

Resources and
Processes Impact on
outcomes attainment

Table 1: Summary of three models of accreditation

15

The Journal of Engineering Education

July - 2013

progress towards its ultimate vision. (See the

process cycle shown in Fig (1).

complex engineering activities, with an

understanding of the limitations.

Graduate Attributes:

6.

The POs formulated for each programme

must be consistentwith the Graduate Attributes
mentioned in Washington Accord. Graduates
Attributes (GAs) form a set of individually
assessable outcomes that are the components
indicative of the graduates potential
to acquire competence to practice at the
appropriate level. The GAs are exemplars of the
attributes expected of a graduate of an
accredited programme. The Graduate Attributes
of the NBA are as following:

The engineer and society: Apply

reasoning informed by the contextual
knowledge to assess societal, health,
safety, legal, and cultural issues
and theconsequent respon-sibilities relevant
to the professional engi-neering practice.

7.

Environment and sustainability:

Understand the impact of the professional
engineering solutions in societal and environmental contexts, and demonstrate the
knowledge of, and need for sustainable
development.

8.

Ethics: Apply ethical principles and

commit to professional ethics and responsibilities and norms of the engineering
practice.

9.

Individual and team work: Function

effectively as an individual, and as a
member or leader in diverse teams, and in
multi-disciplinary settings.

1. Engineering knowledge: Apply the

knowledge of mathematics, science,
engineering fundamentals, and an
engineering specialisation for the solution of
complex engineering problems.
2.

3.

Problem analysis: Identify, formulate,

research literature, and analyse complex
engineering problems reaching substantiated conclusions using first principles
of mathe-matics, natural sciences, and
engineering sciences.

10. Communication: Communicate effectively

on complex engineering activities with the
engineering community and with
t h e society at large, such as, being able
to comprehend and write effective reports
and design documentation, make effective
presentations, and give and receive clear
instructions.

Design/development of solutions:
Design solutions for complex engineering
problems and design system components
or processes that meet the specified needs
with appropriate consideration for public
health and safety, and cultural, societal,
and environmental considerations.

4.

Conduct investigations of complex

problems: Use research-based knowledge
and research methods including design of
experiments, analysis and interpretation of
data, and synthesis of the information to
provide valid conclusions.

5.

Modern tool usage: Create, select, and

apply appropriate techniques, resources,
and modern engineering and IT tools,
including prediction and modelling to

11. Project management and finance:

Demonstrate knowledge and understanding
of the engineering and management
principles and apply these to ones own
work, as a member and leader in a team,
to manage projects and in multidisciplinary environments.
12. Life- long learning: Recognise the need
f o r, a n d h a v e t h e p r e p a r a t i o n a n d
ability to engage in independent and life-

16

The Journal of Engineering Education

July - 2013

long learning in the broadest context of

technological change.

programme. It sums up students achievement

and has no other real use except as a
description of what has been achieved. Thus,
the use of summative assessment enables a
grade to be generated that reflects the students
performance. The present examination system
practiced in our universities falls under
summative assessment. It only evaluates the
student without giving any feedback to him or
his teacher as to how learning-teaching process
has to be modified in future. This is a remnant
of output-based education we have been
practicing so far. With the paradigm shift from
output-based to outcomes-based education, we
will have to turn to formative assessment as a
concomitant part of student-centric education.

Outcomes Assessment:
NBA guidelines clearly state that The
development of different rubrics and the
achievement of the outcomes need to
be clearly stated in the Self Assessment
Report (SAR). The results of assessment of
each PO should be indicated, since they
play a vital role in implementing the Continuous
Improvement process of the programme.
The institute shall provide the ways and means
of how the results of assessment of the POs
help to refine processes of revising/redefining
the POs

Out of twelve graduate attributes listed by

Washington Accord, some attributes like ability
to design or ability to use mathematical
principles can be demonstrated through
conventional examination, oral or written. In other
words, the related outcomes can be assessed
through usual summative assessment
techniques. But there are many others which
can be described as soft outcomes, for example,
problem solving skills, lifelong learning skills
team skills and communication skills. These soft
outcomes cannot be demonstrated or attained
only through summative assessment. Their
formative assessment has to be made in a
manner where performance expectations are
clearly set and actual performance is judged in
an explicit fashion so as to give corrective
feedback to both student and the teacher.
Rubrics are the assessment devices used for
such cases. It may be broadly stated here that
out of 12 Graduate attributes, attribute number
6 (viz.Engineer and Society) onwards can be
termed as soft outcomes and may need rubrics
for assessment. Attributes 1 through 5 are
relatively hard in nature and can be assessed
through examinations and project work.

ABET Definitions released in 2011 define

assessment as one or more processes that
identify, collect and prepare data to evaluate the
attainment of student outcomes. Effective
assessment uses relevant direct, indirect,
quantitative and qualitative measures as
appropriate to the outcome being measured.
Reliable assessment of outcomes is important
not only for recording student performance in a
subject but also for monitoring the overall
teaching-learning process of a program. Based
on the results of the assessment, stakeholders
can take corrective action to ensure that
outcomes are attained to the satisfaction of all.
Assessment is often described in terms of
formative assessment or summative
assessment. Formative assessment has been
described as being feedback assessment for
learning. Formative assessment is usually
carried out at the beginning of a programme or
during a programme. The students performance
on the assessment tasks can help the teacher
to modify the direction of the teaching in order
to help the learning process of the student. Thus
formative assessment is a part of the learningteaching process. On the other hand,
Summative assessment is assessment that
tries to summarize student learning at some point
in time usually at the end of a module or

Rubrics for Outcome Assessment:

Rubrics are a way of explicitly stating what
is the expected student performance in a chosen
17

The Journal of Engineering Education

July - 2013

area of learning. They may lead to a grade or a

score but they are necessarily more specific
and more detailed than a grade or a mark-sheet.
Rubrics imagine a student performance to be
comprised of many levels and provide
descriptors for the performance at each level.
These descriptors help students know what they
need to work on in future so as to improve their
performance. They tend to increase overall
reliability of the assessment process especially
when more than one evaluators are present for
assessing the soft skills. Moreover, they provide
a common language of interaction between
teacher, evaluator, and the students.

game. Round number in the assessment

signifies number of the attempts that have been
put in for the improvement. (please refer Fig. 2).
Types of Rubric:
Rubrics are of two types, holistic and
analytic. The example given in Fig 2 is of analytic
type. It is designed to reflect different levels of
performance constituting a single event. This
design has to be done meticulously. Scale
provided for different dimensions can be further
weighed to indicate their relative importance.
Assessment made using a well designed
analytic rubric can demonstrate progress of the
student over time in some or all dimensions
when used repeatedly. Analytic rubric takes time
to develop and use but it makes formative
assessment of task specific performance skill
possible at individual student level.

Rubrics contain three components:

1.Dimensions (performance criteria) 2.Scale
(levels of performance), and 3.Descriptors.
For example, see the rubric given in Fig 2
that is used in Technical Communications
course (Course No.212387 of 2008 Syllabus))
offered in Petroleum and Petrochemical
Engineering Department of MAEERs
Maharashtra Institute of Technology, Pune,
under University of Pune, for assessing students
oral presentation skills. Using this rubric, the
student is judged for the outcome related to
graduate attribute number 10 viz.
Communication. Copy of this rubric is given to
the class when dates of oral presentations are
announced. Evaluation of the student presenting
is done by the peers in his group. It is interesting
to see that in spite of multiplicity of the evaluators
(typically 12 to 15) hardly any deviation is
reported in the evaluation of any particular
student. This also shows accuracy of the
assessment tool. If deviation in peer group rating
is objectionably large, descriptors or dimensions
or both, can be modified by talking to the group
and removing any element of subjectivity in the
given rubric. Student is encouraged to work on
his presentation till he comes up to at least level
3 in his performance. Since the rules of the game
are known and the assessment fair, students
enthusiastically participate in level improvement

In holistic rubric, evaluator makes his

assessment by forming an overall impression
of a performance and matching it to the closest
description from the sets of descriptors attached
to the scales. Each descriptor on the scale
describes performance on more than one
performance levels. In other words, in holistic
rubric, resolution between different performance
levels is blurred. Thus holistic rubric takes less
time to develop and use. These types of rubrics
are more suitable for course level assessment
rather than individual student level assessment.
Holistic rubrics are often written generically and
can be used with many tasks. Oral presentation
rubric given in Fig 2 will look as in Fig 3, in
holistic form.
Implementation Issues:

18

Teaching faculty is at the heart of

assessment process. Teachers must be
involved in rubrics development where they
can offer insights based on their collective
and cumulative assessment experience.
They may start with holistic rubrics first and
then refine them gradually into meaningful
and workable analytic rubrics. Faculty
consensus on rubric is crucial to the fair

The Journal of Engineering Education

July - 2013

ORAL PRESENTATION RUBRIC

Course Name:
Technical Communications (Course No. 212387, Syllabus 2008. University of Pune. Faculty of Engineering.)

Evaluator: XYZ

Date: dd/mm/yyyy

Student: ABC

Round No.: XX
Scale

Dimension

Score

Organization

Audience cannot
follow the
presentation because
there is no proper
sequence of
information

Audience has
difficulty following
presentation
because student
jump from one
point to the other

Student presents
information in
logical but dry
sequence.
Audience can
follow.

Student presents
information in
logical and interesting
sequence which
motivates the
audience to follow.

Subject
Knowledge

Student cannot
answer any question
about subject
information he has
presented

Student is able to
answer only a few
elementary
questions.

Student answers
most of the
questions, But
fails to elaborate.

Student answers all

the questions with
explanations and
elaboration.

Graphics

Student displays
graphics without
context or no
graphics

Student displays
graphics that rarely
support the text and
presentation.

Student graphics
relate to the text
and presentation.

Student graphics
explain and
complement the text
and presentation.

Students
presentation has five
or more spelling
errors and/or
grammatical errors.

Presentation has
three misspellings
and/or
grammatical
errors.

Presentation has no
more than two
misspellings and/or
grammatical
errors.

Presentation has no
misspellings or
grammatical errors.

Student reads out the

presentation with no
eye contact.

Student
occasionally uses
eye contact, but
still reads most of
the presentation

Student maintains
eye contact most
of the time but
frequently returns
to notes.

student maintains
eye contact
throughout, Does not
return to the notes.

Student mumbles,
incorrectly
pronounces terms,
Cannot be heard
at the back.

Student incorrectly
pronounces terms
Audience members
have difficulty
hearing
presentation.

Student voice is
clear. Student
pronounces most
words correctly.
Most audience
members can hear
presentation

student uses a clear

voice modulation
and correct, precise
pronunciation of
terms so that entire
audience can hear
presentation

Grammar

Eye Contact
with
Audience

Elocution

Total

FIG. 2: Example of Analytic Rubric

19

The Journal of Engineering Education

July - 2013

ORAL PRESENTATION RUBRIC

Course Name:
Technical Communications (Course No. 212387, Syllabus 2008. University of Pune. Faculty of Engineering.)

Evaluator: XYZ

Date: dd/mm/yyyy

Student: ABC

Round No.: XX
Scale

- Audience cannot
follow the
presentation because
there is no proper
sequence of
information.

- Audience has
difficulty following
presentation
because student
jump from one
point to the other.

- Student presents
information in
logical but dry
sequence.
Audience can
follow.

Student presents
information in
logical and interesting
sequence which
motivates the
audience to follow.

- Student cannot
answer any question
about subject
information he has
presented.

- Student is able to
answer only a few
elementary
questions.

- Student answers
most of the
questions, But
fails to elaborate.

- Student answers all

the questions with
explanations and
elaboration.

- Student displays
graphics without
context or no
graphics

- Student displays
graphics that rarely
support the text and
presentation.

- Student graphics
relate to the text
and presentation.

- Student graphics
explain and
complement the text
and presentation.

- Students
presentation has five
or more spelling
errors and/or
grammatical errors.

- Presentation has
three misspellings
and/or
grammatical
errors.

- Presentation has no
more than two
misspellings and/or
grammatical
errors.

- Presentation has no
misspellings or
grammatical errors.

- Student reads out the

presentation with no
eye contact.

- Student
occasionally uses
eye contact, but
still reads most of
the presentation

- Student maintains
eye contact most
of the time but
frequently returns
to notes.

- Student maintains
eye contact
throughout, Does not
return to the notes.

- Student mumbles,
incorrectly
pronounces terms,
Cannot be heard
at the back.

- Student incorrectly
pronounces terms
Audience members
have difficulty
hearing
presentation.

- Student voice is
clear. Student
pronounces most
words correctly.
Most audience
members can hear
presentation

- Student uses a clear

voice modulation
and correct, precise
pronunciation of
terms so that entire
audience can hear
presentation

FIG. 3: Example of Holistic Rubric

20

The Journal of Engineering Education

July - 2013

to both teacher and the student for improvement

at individual student level. While implementing
rubrics, teachers consensus and active
involvement is a must. Since rubrics
development and use is a time consuming
process, due regard has to be given to it when
deciding a teachers load. Initial work inputs and
entry level barriers involved in outcome based
education in general, and rubric development
exercises in particular, are understandably high.
It is hoped that Indian teaching fraternity will not
only adapt itself to outcome based education
system but will also see its own creativity
unleashed in the process.

and reliable outcome assessment.

Development and use of rubrics is a time

consuming process and due regard should
be given to this fact while deciding a
teachers load. If possible, teaching
assistants should be employed to facilitate
the assessment work.

Outcome definition should be precise and

clear before a rubric is developed for its
assessment.

It should be noted that rubrics are not

needed for all the outcomes being assessed.
They are needed only where the assessment
is likely to become more subjective.

References:

While evaluating the data obtained from

assessment process, it should be noted that
not all students are expected to attain all
the outcomes. We are expected to develop
our own and realistic benchmarks for
outcome attainment level.

1.

Ashley Kranov, Michael Milligan, Joe

Sussman, Dispelling Myths: Common
Misconception
about ABET and
Accreditation, ABET Annual Conference,
October 2011

2.

NBA website PPT on Outcome Based

Education and Accreditation Awareness
Workshop for Evaluators and stakeholders;
http://www.nbaind.org/En/1027-forms-andformats.aspx

3.

Gloria Rogers, ABET Fall 2010 Webinar on

Defining Student Outcomes, www.abet.org

4.

Amy Driscoll and Swarup Wood, Developing

Outcomes-based Assessment for Learnercentric Education, Stylus Publishing,
Sterling, Virginia (2007)

Conclusions:
Outcomes-based accreditation has come to
stay in the context of engineering education in
India. Reliable measurement of outcomes
attainment, forms an important part of NBA
documentation. It also provides crucial decision
making points in programmes journey towards
its ultimate vision as reflected in its Programme
Educational Objectives. Rubrics are assessment
devices used in case of soft outcomes. Holistic
rubrics are useful in giving overall feedback at
course level whereas analytic rubrics serve as
instruments of summative assessment useful

21