Data-Driven Design for User Experience

Emi Kwon
UX Lead @MetLife Japan

Why Data-Driven Design
for User Experience?

Why Data-Driven Design for User Experience?
Data-Driven Design for User Experience

The decision fatigue
What if I was wearing a face mask?
What if I was not wearing a face mask?
What if________?
We make some ###
decisions every day
…and a lot of it is based
on ### hypotheses

Hypothesis >> Causal data >> Optimized decision (design!)
(a) Hypothesis + (b) causal data + (c) decision =
change with an impact (with less decision fatigue)

Data ecosystem — Data lake, data warehouse…what does it mean for UX?

Small data and big data — The opportunities and pitfalls

Research method basics — Qualitative, quantitative or triangulated

User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents

Unstructured data
Semi-structured
data
Structured
data
Data that has no inherent structure
and is usually stored as different
types of files

E.g. Text documents, images, videos
Textual data files with an apparent
pattern, enabling analysis

E.g. JSON and XML files
Data having a defined data model,
format and structure

E.g. Database
Data structure

Data lake vs data warehouse

Data lake vs data warehouse + data mart

Not Customer Facing Customer Facing
Sales App 1
Sales
Engagement tool
TTL admin tool
Sales App 2
New Wave pilot
ATS
Post-Transaction
tool
Sales Website
New Wave for
Customer
Transaction
App
Customer App2
Smart Simulator
Marketing Website
Video Chat Chatbot
Customer
Micro-site
Customer App 1
Sales
Customer Sales Admin
New Wave App
New Wave Web
Pilot
Existing Proposed Active Inactive
SimplePath
PIC Kim
PIC John
PIC James
PIC Wendy
PIC Jane PIC Mark
PIC Bryan
PIC Jane PIC Jack
PIC Henry
PIC Tom PIC Anthony
PIC TBC PIC Wendy
Mapping out your data ecosystem can be like…
PIC TBC
PIC Henry
PIC Jack
PIC TBC
PIC Annie PIC TBC

Gender Age Group
Per Page
Use
Time Per
Page
# Logins
# Login
Errors
#
Registrat’n
# Reg
Completed
# Reg
Bailed
Avg Reg
Time
Sales App 1
Now: ✖
Future: ⃝
Now: ✖
Future: ⃝
⃝ ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ ＊
Sale App 2 △ △ ⃝ ⃝ ⃝ ⃝ ✖ ✖ ✖ ✖
Sales App 3 △ △ ⃝ ⃝ ⃝ ⃝ ? ? ? ?
SmartPath ? ? ⃝ ⃝ ? ? ✖ ✖ ✖ ✖
NewWave
App 1
? ? ⃝ ⃝ ? ? ✖ ✖ ✖ ✖
NewWave
App 2
△ △ ⃝ ⃝ ? ? ? ? ? ?
NewWave
Web 1
△ △ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
NewWave
Web 2
? ? ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
Admin tool 1 ✖ ✖ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
Admin tool 2 ✖ ✖ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
△ : should, but no defined requirement as yet ＊ : for individual pages or for registration as a whole?
Mapping out your data attributes can be like…

Q: What do you want to get from doing data-driven design? ✨
1) To understand my users better
2) Data is the new oil; no way I can do design without being data-driven

3) To add something data-driven to my portfolio/ work experience
4) Not sure yet; will see what I can learn and go from there

5) Others (Feel free to write your answer in the chat 🙂)

The data fatigue..
So much data is generated.
So much data is recorded.
So much data is accessible.
In this data lies many
many stories…and among
them, many many stories
of users…
Analysis paralysis..

https://www.youtube.com/watch?v=TzxmjbL-i4Y
Big Data!!!

Goals, Location, Data structure, Data
preparation, Longevity, Reproducibility…
Volume, Velocity, Variety
+ Value and Veracity
3 ways big data is different from small data

Big data and small data… what does it mean for UX design?
https://www.kaushik.net/avinash/best-web-analytics-tools-quantitative-qualitative/
Call center log
Call center log , Sales and revenue data

Data vs Insight
Insight
Organized, structured, categorized,
useful, condensed, calculated
Individual facts, figures, signals,
measurements
+ Context
+ Meaning
+ Perspective
Data
Information
Knowledge
Understanding, integration, applied,
reflected upon, actionable, accumulated,
principles, patterns, decision-making
process
Idea, learning, notion, concept,
synthesized, compared, thought-
out, discussed
Decision
risk
Value
Volume,
velocity
&
variety

Data
lake
>>
Data
warehouse
>>
> UX Data mart >> Data-driven design decisions
The way we (would like to) fertilize our design…

Call center log
Call center log
What if only big data exists?
Usage data
Only big data ? Call center log

Call center log
Call center log
Behavioral insights
Only small data ?
What if only small data exists?

What if only small or big data exists?
You believe you know what users want… Without knowing what their critical contexts are….

#multiplicity
“The quest for a "single source of the truth" on the web is futile…
…the quest for a single tool/source to answer all your questions will ensure that your
business will end up in a ditch, and additionally ensure that your career (from the
Analyst to the web CMO) will be short-lived… |

https://www.kaushik.net/avinash/big-data-imperative-driving-big-action/

Multiplicity of data leads to good perspective-taking
You know the real risk lies in somewhere not so visible… So you will continue to look out for more evidence

Q: Is big data important in doing your work
(design, production, development…)? ✨
1) A lot; using big data can make or break a deal in my line of work

2) To some extent; it’s good to have big data around

3) Hm…. might be, but no specific use case yet

4) I don’t care for big data
5) Never a deal-breaker


Big data and small data — The opportunities and pitfalls


Q&A and Discussion
Contents

26
User research & test methods
Why What

Qualitative research_Interview flow
• Sampling & Participant recruiting
• Drafting questionnaire
• Data collection • Data analysis
• Survey report
• Cross-functional action plan
• Development
• Conduct interview
• Team de-brief

Qualitative analysis tools
Tag and organize your interview notes https://www.aureliuslab.com/ https://evernote.com/

29
Qualitative analysis_Thematic analysis & Word cloud mapping

• Samping & Participant recruiting
• Survey report
• Report share-out
• Building survey form
• Distribution of the survey
30
• Development
Survey 2.0 https://www.qualtrics.com/
Quantitative research_Survey flow
https://www.jotform.com/table-templates/rainbow-sheet
Survey 1.0 https://www.surveymonkey.com/

31
Quantitative analysis_Describe, Relate, Regression, PivotTable
https://www.qualtrics.com/support/stats-iq/analyses/describe-data/

32
Quantitative analysis tools

User analytics…what about it?
Dimension
• Content information
• Conversion funnels
• Navigation paths
• Visitor details
Strategy
• Acquisition
• Adoption
• Engagement
Metrics
• # of users and sessions
• Average session duration
• Average pages per session
• New/ Returning visitors
• Bounce Rate
• …
34
Dimensions
Metrics
Flow visualization
Adobe Analytics dashboard

User analytics…what can they do?
• Real-time events
• Heatmaps
• Session replay videos
• A/B testing

Building it together and socializing your UX data

A/B testing
What do to
• Getting a testing tool
• Define your UX success metrics
• Define hypothesis to be tested
• Define variables and learning
• Create the variables
• Measure and activate
• Analyze the results
• Document learning & Evangelize
• Ideate & Repeat
https://www.optimizely.com/

https://www.convertize.com/ab-test-significance/

https://business.adobe.com/products/target/adobe-target.html
Control Treatment

https://www.optimizely.com/insights/blog/how-obama-raised-60-million-by-running-a-simple-experiment/
How Obama raised $60 million
by running a simple A/B test
6 Media + 4 Button = 24 combinations.. 40.6% lift in sign-up rate

Numbers can inform you of
what is happening,
but they won’t tell you
why something is happening.
User analytics and A/B testing_The pitfalls
#multiplicity
Now you need to dig deeper with qualitative data.


Small data and big data — The opportunities and pitfalls


Q&A and Discussion
Contents The why

https://bootcamp.uxdesign.cc/remote-usability-test-heres-all-the-tools-you-need-403717336552

Remote usability testing
• Sampling & Participant recruiting
• Survey report
• Building survey form
• Distribution of the survey
• Development
• Conduct interview
• Team de-brief

Mon Tue
8/10
8/17
8/24
8/31
9/7
9/14
8/11
8/18
8/25
9/1
9/8
9/15
8/12
8/19
8/26
9/2 Emi OOO
9/9
9/16
8/13
8/20
8/27 Emi OOO
9/3
9/10
9/17
8/14
8/21
8/28
9/4
9/11
9/18
Tue Wed Thu Fri
Test scenario (E)
Eye-tracking hands-on (S)
Eye-tracking concept study (W)
Eye-tracking test (E, J, H)
Data-driven UX
monthly call

w/ Biz, Data Science
Test scenario (E, H) Test mockup (H) & Compliance Form (E)
Executive summary (E, M)
Test analysis

BA sharing (J)
Data collection (H)
Compliance Form (E)
User test (E, W)
Data collection (E, W)
Thematic analysis (E) Test analysis

BA sharing TBC (E)
Demo drill (E,S)
Projects overview
Sales App 1 (Emi, Tom)

An app description is an app store optimized
product definition. It greatly influences your
product's success. There are three components
of an app's 'definition': its Name, its Description
in the marketplace, and Screenshots.
Sales App 2 (James, Hannah)

product's success. Continuous touchpoint:
whenever you need, you can get it done, fast
and easy.
NewWave Website pilot (Michael, Wendy)

product's success. Continuous touchpoint:
whenever you need, you can get it done, fast
and easy.
Test scenario TBC
Eye-tracking
demo w/ DevOps
Data-driven design calendar

But don’t be a user test ninja…
Every day A/B test
with 6 design variants plus…
Try all these methods this and next month…
No, you can’t! It will just show
your lack of confidence!
https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Dunning-Kruger Effect
Time will come
when you hit
this mark…
Every day A/B test
with 6 design variants plus…
ry all these methods this and next month…
https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Make informed decisions with causal data
What if I was wearing a face mask?
What if I was not wearing a face mask?
What if________?
We make some ###
decisions every day
…and a lot of it is based
on ### hypotheses

Q&A and Discussion

Data-Driven Design for User Experience

Related slideshows

More Related Content

Data-Driven Design for User Experience