The document discusses the lean analytics cycle of metrics, hypothesis, experiment, and act. It provides examples of how Hello Bar used this process to improve their installation rate. They found a low installation rate in metrics, hypothesized that more options would increase installations, tested this in an experiment, and achieved a 40% increase. Through dozens of experiments, their rate increased by 89%. The document encourages analyzing metrics to find opportunities, forming hypotheses through research, rigorously testing hypotheses, and making data-driven decisions.
Report
Share
Report
Share
1 of 96
Download to read offline
More Related Content
Lean Analytics Cycle
1. Lean Analytics
Lean Product & Lean UX Silicon Valley - July 2014
Hiten Shah
hnshah@gmail.com
Download the slides: h p://kiss.ly/leanprod2014
32. Hello Bar Experiment Hypothesis
If
more
installa2on
op2ons
are
given,
then
more
people
will
add
Hello
Bar
to
their
website
due
to
the
need
for
a
WordPress
plugin
and
ability
to
email
the
code
to
someone
else.
33. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next
39. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next
40. Full funnel metrics for the experiment
Variation
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed
Totals 7456 906 342 329 276
% prev step 12.15% 37.75% 96.20% 83.89%
% drop off -87.85% -62.25% -3.80% -16.11%
% of visitors 100% 12.15% 4.59% 4.41% 3.70%
Control
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed
Totals 7469 925 353 341 197
% prev step 12.38% 38.16% 96.60% 57.77%
% drop off -87.62% -61.84% -3.40% -42.23%
% of visitors 100% 12.38% 4.73% 4.57% 2.64%
41. Resulted in a 30% installation rate increase
HTTP://ISVALID.ORG
Experiment resulted in a 40% improvement
42. ACer
a
dozen
more
experiments,
our
installa2on
rate
is
now
hovering
between
5%
and
7%
That’s
an
addi;onal
89%
increase!!!
The Lean Analytics Cycle At Work
53. “A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data you have to base your
hypothesis on, the be er it will be.”
— Michael Aagaard, h p://kiss.ly/abthypo
74. Only
1
out
of
5
tests
win.
4
out
of
5
tests
cost
you
money.
$
75. 1)
Which
top
metric
are
we
trying
to
improve?
2)
What
can
you
anchor
your
metric
against?
3)
What
is
your
hypothesis
about
why
this
metric
is
at
x%?
4)
What
do
we
expect
from
this
experiment?
5)
What
was
the
result
and
what
did
we
learn
from
it?
HTTP://KISS.LY/6QSAB
Answer key questions for each experiment
84. KISSmetrics Experiment Hypothesis
If
people
authen2cate
with
Google,
then
more
people
will
sign
up
due
to
their
exis2ng
familiarity
with
Google
Analy2cs
and
alignment
with
our
winning
headline.
95. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next
96. Hiten Shah
hnshah@gmail.com
@hnshah
You will get all you want in life if you help enough
other people get what they want.
Zig Ziglar
Download the slides: h p://kiss.ly/leanprod2014