Lec11 ch8
Lec11 ch8
Lec11 ch8
Lecture 11
1
Lecture Outlines
• Evaluation Criteria
❖ Usability (Quantitative and Qualitative Measures)
❖ UX
• Evaluation Methods
❖ Focus Interview/Observation Study
❖ Expert/Heuristic Reviews
❖ Measurement
❖ Surveys
• Homework 8
2
Introduction
• Designers can become so entranced with their creations that they may
fail to evaluate them adequately.
– Priorities change
3
Evaluation Criteria
4
Evaluation Criteria: Usability
Usability:
• Usability refers to the ease of use and learnability of the user interface.
https://www.nngroup.com/articles/quant-vs-qual/
5
Evaluation Criteria: Usability - Quantitative Measures
Quantitative (quant)
• offer an indirect assessment of the usability of a design. They can be
based on users’ performance on a given task (e.g., task-completion times,
success rates, number of errors) or can reflect participants’ perception of
usability (e.g., satisfaction ratings).
6
Evaluation Criteria: Usability - Quantitative Measures
• For example, suppose we would like to test a new motion-based interface for a
smart phone game. We could have a pool of subjects play the game, using both
the conventional touch-based interface and also the newly proposed motion
based one. We could compare the score and assess the comparative effectiveness
of the new interface.
Qualitative (qual)
– Consist of observational findings that identify design features easy or hard to
use
9
NASA TLX (Task Load Index) is one of the often used
semi-standard questionnaires for this purpose [NASA].
NASA Task Load Index method assess work load on 7
point scales. Increments of high, medium and low Excerpts from the IBM Usability Questionnaire for
estimates for each point result in 21 gradations on the computer systems [IBM].
scale 10
Evaluation Criteria: UX
• It is also deeply related to the user’s emotion and perception that result from
the user or anticipated use of the application (through the given interface).
11
Evaluation Criteria: UX – Cont.
• A distinction can be made between usability methods that have the objective of
improving human performance,
and user experience methods that have the objective of improving user
• Note that the notion of UX encompasses usability, that is, “usually” high UX
translates to high usability and high emotional attachment.
12
User Experience
Usability,
Value, Function,
Emotion/Affect, Size, Weight,
Expectation, Preference,
Prior experience, Satisfaction,
Physical fitness, Aesthetics,,
Personality, Interaction Reputation,
Motivation, User Product
Adaptability,
Skill, Mobility, etc.
Age, etc.
Social Context of
Factor Usage
Cultural
Factor
Time pressure, Time, Place,
Peer pressure Gender, Trend, Weather,
Social status, Rules, Language, Public/Private usage,
Social obligations, Norm, Standards, etc.
etc. Religion, etc.
13
Figure 8.3 Various aspects to be considered in totality for assessing user experience (UX).
Evaluation Methods
• A given method may be general and applicable to many different situations and
objectives, or more specific and fitting for a particular criterion or usage
situation.
– Type and number of evaluators (e.g. several HCI experts vs. 100’s of domain
users)
15
Focus Interview / Observation Study
17
Focus Interview / Observation Study
– In this case, the technique is focused on investigating for any gap between
the interaction model of the system and that of user.
– We can deduce that cognitive walkthroughs are fit for relatively the earlier
stage of design, namely interaction modeling or interface selection (vs.
specific interface design).
18
Figure 8.4 Interviewing a subject upon simulating the usage of the interface with a mock-up
19
… I expect to see … Please tell
the list of latest me what is
horror movies and going
will try to select it through your
with a mouse … mind as is …
20
Focus Interview / Observation Study
• Note that the interview/simulation method, due to its simplicity, can be used not
only for evaluation but also for interaction modeling and exploration of
alternatives at the “early” design stage.
• We have already seen design tools such as storyboards and wire-framing which
can be used in conjunction with users or experts for simultaneous analysis and
design.
21
Summary
Evaluators / Size Actual users / Medium sized (10~15)
Type of evaluators Focused (e.g. by expertise, age group, gender, etc.)
Formality Usually informal (not controlled experiment)
Timing and Stage Objective Enactment Method
Objectives Early Interaction model and flow Mock-up /
Wizard of Oz
Middle Interface selection Mock-up /
Wizard of Oz
Partial simulation
22
Expert/Heuristic Reviews
• The difference is that the evaluators are HCI experts and the analysis is carried out
against a pre-prepared HCI guideline, hence called heuristics.
– For instance, the guideline can be general or more specific, with respect to
application genre (e.g. for games), cognitive/ergonomic load, corporate UI
design style (e.g. Android UI guideline), and etc.
23
Expert/Heuristic Reviews
1. Visibility of system status: The system should always keep users informed about
what is going on, through appropriate feedback within reasonable time.
2. Match between system and the real world: The system should speak the users’
language, with words, phrases, and concepts familiar to the user, rather than system
-oriented terms. Follow real-world conventions, making information appear in a
natural and logical order.
3. User control and freedom: Users often choose system functions by mistake and will
need a clearly marked “emergency exit” to leave the unwanted state without having
to go through an extended dialogue. Support undo and redo.
24
Expert/Heuristic Reviews
4. Consistency and standards: Users should not have to wonder whether different
words, situations, or actions mean the same thing. Follow platform conventions.
5. Error prevention: Even better than good error messages is a careful design that
prevents a problem from occurring in the first place. Either eliminate error-prone
conditions or check for them and present users with a confirmation option before
they commit to the action.
6. Recognition rather than recall: Minimize the user’s memory load by making objects,
actions, and options visible. The user should not have to remember information
from one part of the dialogue to another. Instructions for use of the system should
be visible or easily retrievable whenever appropriate.
25
Expert/Heuristic Reviews
8. Aesthetic and minimalist design: Dialogues should not contain information that is
irrelevant or rarely needed. Every extra unit of information in a dialogue competes
with the relevant units of information and diminishes their relative visibility.
9. Help users recognize, diagnose, and recover from errors: Error messages should be
expressed in plain language (no error codes), precisely indicate the problem, and
constructively suggest a solution.
10. Help and documentation: Even though it is better if the system can be used without
documentation, it may be necessary to provide help and documentation. Any such
information should be easy to search, be focused on the user’s task, list concrete
steps to be carried out, and not be too large.
26
Expert/Heuristic Reviews
• A few (typically 3~5) UI and domain experts are brought in to evaluate the UI
implementation in the late stage of the development or even against a finished
product.
• The disadvantage of the expert review is that the feedback from the user is absent
as the HCI expert may not understand the needs of the actual users.
– Even experienced expert reviewers have great difficulty knowing how typical
users, especially first-time users will really behave.
27
Summary
Easy and quick, but prior heuristics assumed to exist and no actual user feedback reflected
28
Measurement
29
Figure 8.6 The initial (left) and redesigned (right) “play” activity/layer for No Sheets: The
new design after evaluation uses a landscape mode and fewer primary colors. The icons for
fast-forward and review are changed to the conventional style, and the current tempo is
shown on top.
30
Surveys
• On the other hand, numerical scores can be obtained from surveys.
• Surveys are used because many aspects of usability or user experience are based on user
perception which is not directly measurable.
• However, answers to user perception qualities are highly variable and much more susceptible
to user’s intrinsic backgrounds.
• Even though the result of the survey is a numerical score, the nature of the measurement is
still qualitative because survey questions usually deal with user perception qualities.
31
Online Surveys
• Online surveys avoid the cost of printing and the extra effort needed
for distribution and collection of paper forms.
Use an odd level scale, 5 or 7 Research has shown odd answer levels with mid
(or Likert Scale) value with 5 or 7 levels produces the best results.
Make questions compact and Questions should be clear and easy to understand. If
understandable difficult to convey the meaning of the question in
compact form, the administrator should verbally
explain.
Give subjects compensation Without compensation, subjects will not do one’s best
or perform the given task reliably.
Categorize the questions For easier understanding and good flow, questions of
the same nature should be grouped and answered in
block, e.g. answer “ease of use” related questions,
then “ease of learning” and so on.
33
Table 8.4 Guidelines for a Good Survey
Safety and Ethics in Evaluation
• Most HCI evaluation involves simple interviews and or carrying out simple tasks
using paper mock-ups, simulation systems, or prototypes. Thus, safety problems
rarely occur.
• However, precautions are still needed. For example, even interviews can become
long and time consuming, causing the subject to feel much fatigue.
• Some seemingly harmless tasks may bring about unexpected harmful effects, both
physically and mentally.
• Therefore, evaluations must be conducted on volunteers who have signed consent
forms. Even with signed consents, the subjects have the right to discontinue the
evaluation task at any time.
• The purpose and the procedure should be sufficiently explained and made
understood to the subjects prior to any experiments. Many organizations run what
is called the Institutional Review Board (IRB), which reviews the proposed
evaluative experiments to ascertain safety and the rights of the subjects. It is best
to consult or obtain permission from the IRB when there is even a small doubt of
some kind of effect to the subjects during the experiments.
34
Summary
• We have looked at various methods for evaluating the interface at different stages
in the development process. As already emphasized, even though all the required
provisions and knowledge may have been put to use to create the initial versions
of the UI, many compromises may be made during the actual implementation,
resulting in a product somewhat different from what was originally intended at the
design stage.
• It is also quite possible that during the development, the requirements simply
change. This is why the explicit evaluation step is a must and, in fact, the whole
design-implement-evaluate cycle must ideally be repeated at least a few times
until a stable result is obtained.
35
Homework 8
• Complete HW8 on BB
• Due Thursday October 26th - 11:59 pm
• Individual homework
• 2 marks
36