USABILITY

TESTING
METHODS
PERTEMUAN 7 – EVALUASI UI/UX

TIM DOSEN EPAP dan DPB

METHODS
Usability Inspection Usability Testing
Reviewer Expert User (end-user)
Findings Supplement Primary
Data Mostly qualitative Qualitative or quantitative
or combination of qual. &
quant.
Stage Throughout design life- Tangible product provided
cycle
Methods • Cognitive Walkthrough • Usability Testing
• Heuristic Evaluation
Usability Testing (UT)
Overview
• Usability testing refers to evaluating a product or
service by testing it with representative users.
• Usability testing involves observing users while they
perform tasks with a hardware or software system.
• The goal is to identify any usability problems, collect
qualitative and quantitative data and determine the
participant's satisfaction with the product.

https://www.usability.gov/how-to-and-tools/methods/usability-testing.html
User Testing vs Usability Testing
User Testing vs Usability Testing
When should you do:
• User Testing: Right after you have got the idea.
• Usability Testing: As soon as you have drawn those
random hand drawn sketches on paper
What to expect from:
• User Testing: Do people need the solution I am
working on?
• Usability Testing: Can people easily use the app ( for
a given set of task) I am working on?

http://blog.canvasflip.com/index.php/2016/06/12/user-testing-vs-usability-testing/
User Testing vs Usability Testing
What to ask during:
• User Testing:
◦ How do you currently do <problem you are solving>?
◦ Did you even think of a better way of doing this?
◦ Will you like to do this task <the way your solution works>?
◦ Will you pay money (for transaction apps) / share content for this
solution?

• Usability Testing:
◦ Can you try doing this <new way of solution>?
◦ How would you like to login to this solution?
◦ Can you get <a small task in your solution> done in 10 seconds?

http://blog.canvasflip.com/index.php/2016/06/12/user-testing-vs-usability-testing/
Advantages
• You can get feedback that reveals possible design flaws and
other issues.
• You can get reliable measures of usability (see summative
usability testing).
• Experienced test facilitators can elicit feedback from users to
help understand why they had problems.
• Low and medium-fidelity prototypes are cost-effective to test.
• It is easy to have project manager and developers as observers.
• You can produce video clips from test sessions to show
problems.

http://www.usabilitybok.org/usability-testing
Disadvantages
• Not all problems will be found with small samples of users.
• You may not have access to users that match the user profile.
• Not all tasks may be "right" for all users.
• Lab testing takes users away from their natural work
environment.
• Technical setup may be complex and require domain experts
and additional time for setup and debugging.
• An inexperienced facilitator can influence the results by using
too many hints, asking biased questions, or providing nonverbal
cues about the tasks.

http://www.usabilitybok.org/usability-testing
Procedure
Usability testing consists of five primary phases:
1. Planning
2. Pretest or pilot
3. Test sessions
4. Post-test or debrief
5. Analysis, interpretation, and presentation of the
results.

http://www.usabilitybok.org/usability-testing
Materials Needed
In general, the materials needed to run a usability test
include:
• The system (paper sketch, model, display mockup,
software, website)
• Physical or portable test lab (camera setup, observation
room)
• Written test materials (informed consent, questionnaires,
task scenarios, observation data sheets)
• Technical setup (servers, "live" or simulated test data)
• Connections for remote observers

http://www.usabilitybok.org/usability-testing
Procedure (1)
Planning
◦ Scope: Indicate what you are testing
◦ Purpose: Identify the concerns, questions, and goals for this
test.
◦ Schedule & Location: Indicate when and where you will do
the test.
◦ Sessions: Describe the sessions taken.
◦ Equipment: Indicate the type of equipment you will be using
in the test.
◦ Participants: Indicate the number and types of participants to
be tested you will be recruiting. >>More<<

https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html
Procedure (1)
Planning
◦ Scenarios: Indicate the number and types of tasks included in
testing.
◦ Metrics: Subjective metrics, include the questions you are
going to ask the participants. >>More<<
◦ Quantitative metrics: Indicate the quantitative data you will be
measuring in your test. >>More<<
◦ Roles: Include a list of the staff who will participate in the
usability testing and what role each will play.

https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html
Participant
• Usability Testing → representative users
• Number of participant:
◦ Norman Nielsen: 5 users (qual.) or 20 users (quant.)
>>more<<
◦ Janet M. Six and Ritch Macefield: >>more<<
Problem-discovery study: 5-10 users
Comparative study: 10-12 users
Num of Participant (Nielsen)

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
Num of Participant (Nielsen)

https://www.nngroup.com/articles/quantitative-studies-how-many-users/
Num of Participant (Six & Macefield)

https://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-participants-for-usability-studies.php
Metrics
Effectiveness
• Accuracy → Error rate
• Completeness → Successful task completeness
Efficiency (resources expended)
• Time → Time on task
Satisfaction (comfort and acceptability)
• comfort and acceptability questionnaire
Data Gathered
Quant. Qual.
Data related to: Data related to:
• Success rates • Observations about pathways
• Task time participants took
• Error rates • Problems experienced
• Satisfaction questionnaire • Comments/recommendations
ratings • Answers to open-ended questions
Procedure (1)
Planning
◦ Usability Testing Plan Template:
https://www.usability.gov/how-to-and-
tools/resources/templates/usability-test-plan-template.html
Procedure (2)
Pre-test or Pilot Testing
• Conduct pilot tests with internal users to debug
instructions and tasks, verify that the hardware and
software are working, and determine if there is
adequate time for the session.
• Resolve any technical or logistical problems with the
test plan and setup. Fix any problems with written
test materials.
• Finalize the schedule and send it to all the observers.
Procedure (3)
Test Session
1. Welcome the user
Brief the user. Sign informed consent form(s), and the nondisclosure
agreements (NDAs) if needed,

2. User fill out pretest questionnaire

Which can be used to verify screening information and gather
additional background information.

3. Observing users while they perform tasks

Do not give hints or assistance unless necessary. Record time spent
on each task, errors and problems user may face

http://www.usabilitybok.org/usability-testing
Moderating Technique (1)
Some common moderating techniques include:
• Concurrent Think Aloud (CTA) is used to understand
participants’ thoughts as they interact with a product
by having them think aloud while they work. The goal
is to encourage participants to keep a running
stream of consciousness as they work.

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (1)
Techniques Pros Cons
Concurrent • Understand participants’ • Can interfere with usability
Think thoughts as they occur and as metrics, such as accuracy
Aloud they attempt to work through and time on task
(CTA) issues they encounter
• Elicit real-time feedback and
emotional responses

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (2)
Some common moderating techniques include:
• In Retrospective Think Aloud (RTA), the moderator
asks participants to retrace their steps when the
session is complete. Often participants watch a
video replay of their actions.

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (2)
Techniques Pros Cons
Retrospective • Does not interfere with • Overall session length
Think Aloud usability metrics increases
(RTA) • Difficulty in remembering
thoughts from up to an
hour before = poor data

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (3)
Some common moderating techniques include:
• Concurrent Probing (CP) requires that as
participants work on tasks—when they say
something interesting or do something unique, the
researcher asks follow-up questions.

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (3)
Techniques Pros Cons
Concurrent • Understand participants’ • Interferes with natural thought
Probing thoughts as they attempt process and progression that
(CP) to work through a task participants would make on
their own, if uninterrupted

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (4)
Some common moderating techniques include:
• Retrospective Probing (RP) requires waiting until the
session is complete and then asking questions
about the participant’s thoughts and actions.
Researchers often use RP in conjunction with other
methods—as the participant makes comments or
actions, the researcher takes notes and follows up
with additional questions at the end of the session.

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Moderating Technique (4)
Techniques Pros Cons
Retrospective Does not interfere with Difficulty in remembering =
Probing (RP usability metrics poor data

https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
Procedure (4)
Post-test or Debrief
1. User fill out post-test questionnaire
At the end of the session, ask the user to complete a post-test
questionnaire.

2. Interview the user

to confirm they are representative of the intended user group, to gain
general opinions, and to ask about specific problems encountered.

3. Assess the results

Assess the results of the results of quantitative metrics, such as
accuracy and completeness.

http://www.usabilitybok.org/usability-testing
Kuesioner
• Teknik pengumpulan data dengan memberikan
sejumlah pertanyaan yang sudah pasti diberikan
dan dalam bentuk tercetak.
• Pertama yang harus dilakukan evaluator adalah
menentukan tujuan dari kuesioner : informasi apa
yang akan diambil ?
• Keunggulan metode ini yaitu cepat, dapat mencapai
sasaran untuk kelompok pengguna yang besar dan
dapat dianalisis dengan teliti, sedangkan
kekurangannya kurang fleksibel.
Jenis Kuesioner
• General,
pertanyaan untuk mengetahui latar belakang
pengguna dan memasukannya ke dalam subyek
populasi.
• Open-ended,
pertanyaan yang membutuhkan jawaban dan opini
sendiri yang berguna untuk mengumpulkan
informasi subyektif secara umum tetapi sulit untuk
dianalisis.
Jenis Kuesioner
• Scalar,
user diminta untuk memutuskan pernyataan
tertentu berdasarkan skala angka, biasanya
berhubungan dengan setuju atau tidak setuju
terhadap pernyataan yang diberikan.
Jenis Kuesioner
• Multi-choice,
responden diberikan pilihan respon secara eksplisit
dan hanya boleh memilih satu saja atau sebanyak
yang diperlukan.

Misal : Software apa yang pernah Anda gunakan ?

Jenis Kuesioner
• Ranked,
mengurutkan item dalam daftar dan berguna untuk
mengindikasikan preferensi user.

Misal : rangkinglah kegunaan dari metode perintah

Popular Usability Questionnaire
1. System Usability Scale (SUS)
2. USE Questionnaire: Usefulness, Satisfaction, and
Ease of use
3. Software Usability Measurement Inventory (SUMI)
4. Post-Study System Usability Questionnaire (PSSUQ)
5. Purdue Usability Testing Questionnaire
Procedure (5)
Report the results
1. Background Summary: Include a brief summary
including what you tested (website or web
application), where and when the test was held,
equipment information, what you did during the test
(include all testing materials as an appendix), the
testing team, and a brief description of the
problems encountered as well as what worked well.

https://www.usability.gov/how-to-and-tools/methods/reporting-usability-test-results.html
Procedure (5)
Report the results
2. Methodology: Include the test methodology,
explain how you conducted the test.
3. Test Results: Analyze the data recorded.
4. Findings and Recommendations: List your findings
and recommendations using all your data
(quantitative and qualitative)

https://www.usability.gov/how-to-and-tools/methods/reporting-usability-test-results.html
Report Template
Usability Test Report Template:
https://www.usability.gov/how-to-and-
tools/resources/templates/report-template-usability-
test.html
Further Reading

Akses URL:
https://drive.google.com/file/d/1tmbsFAkij
gCyYDf8FHrjsA8F4V0CGb4X/view
(Wajib Email UB)
Video User Research
• https://www.youtube.com/watch?v=Y5LZDaOxhco
Recommended Paper/Article
• https://www.researchgate.net/publication/2286093
27_A_Comparison_of_Questionnaires_for_Assessin
g_Website_Usability
• https://measuringu.com/10-things-sus/
• https://uxpajournal.org/wp-
content/uploads/sites/7/pdf/JUS_Sauro_Feb2015.p
df