GUIDE

THE DEFINITIVE GUIDE TO

A/B TESTING
MISTAKES
AND HOW TO AVOID THEM
 TABLE OF CONTENTS

0 INTRODUCTION 5

1 7 BEGINNER MISTAKES 7

2 YOU ARE STOPPING YOUR TESTS TOO EARLY 21

3 WARNING! IS THE WORLD SABOTAGING YOUR A/B TESTS? 31

4 ARE YOU MISINTERPRETING YOUR RESULTS? 43

5 YOUR BRAIN IS YOUR WORST ENEMY 55

 INTRODUCTION

Unless you’ve been living under a Because you can screw up in so

rock, you know that A/B Testing can many ways it will make you dizzy, and
be extremely beneficial for all your because maths are involved.
marketing and design decisions.
But fear not, we got your back.
It allows you to know exactly what
works, what doesn’t, to test before In this ebook, we’ll look into about
implementing anything so you don’t every mistake you could make while
lose money, and more. A/B Testing.

You’re most likely already doing A/B So grab a cup of coffee (or 3) and let’s
tests. begin!

All the power to you right? Well, not so

fast. Here is the bleak truth…

A staggering number of people doing

A/B tests get imaginary results. We
see it among our clients, and 5 minutes
on Google will tell you the same story.

How come?

A/B TESTING MISTAKES AND HOW TO AVOID THEM 5

 1
7 BEGINNER
MISTAKES

7
 7 BEGINNER MISTAKES

Let’s start off with 7 mistakes most beginners make when A/B testing:

1. They start with complicated tests 4. They don’t prioritize their tests
2. They don’t have a hypothesis for 5. They don’t optimize for the right
each test KPI’s
3. They don’t have a process and a 6. They ignore small gains
roadmap 7. They’re not testing at ALL times

1. They start doing A/B Testing with complicated tests

For your first tests ever, start Starting with A/B Testing is a lot like
simple. Being successful at A/B starting weight training seriously.
Testing is all about process. So it’s You don’t start with your maximum
important that you first go through the charge and complicated exercices.
motions. It would be the best way to injure
yourself badly.
See how theory computes with reality, You start light, and you focus 100% of
what works for you and what doesn’t. your attention on the movement itself
Where you have problems, be it in the to achieve perfect form, with a series of
implementation of the tests, coming up checkpoints to avoid all the ways you
with ideas, analyzing the results, etc… get injured or develop bad habits—
that’ll end up hurting in the long run.
Think about how you’ll scale your
testing practice, or if you’ll need new By doing that, you’ll imprint it in your
hires for example. muscle memory so when you need
to be focused on the effort itself, you
won’t even have to think about the
movement. Your body will instinctively
do it.

8 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

Then and only then, can you achieve So, you got the message: doing
the highest performance possible complicated tests right off the bat
without screwing up. could kill your efforts in the egg.

Exact. Same. Thing. With. A/B Testing. You could be overwhelmed, get fake
results and get discouraged.
You start simple, focus all your
attention on each step of the
process, set up safeguards and
adjust as you go so you don’t have to Here are a couple examples of
worry about it later. things you could test to start
with:
Another benefit if you start with simple
tests is that you’ll get quick wins. • Test copy on your offers,
product pages, landing
Getting bummed out when your first pages (Make it focused on
tests fail (and most do, even when benefits not features, be
done by the best experts out there) is sure that what you mean is
often why people give up or struggle crystal clear)
to convince their boss/colleagues that
split testing is indeed worth the time • Removing distractions on
and investment. key pages (Is this slider
really necessary, or all these
Starting with quick wins allows extra buttons?)
you to create momentum and rally
people to the practice inside your
team/company.

VERSION A VERSION B

BUY ADD TO
VS BASKET

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE 9

2. They don’t have a hypothesis for each test

Every test idea should be based

on data, articulated through an Here are several sources to
informed hypothesis with an build your hypothesis on:
underlying theory. If you’re not doing
this, you’re mostly shooting in the dark • Analytics
without even knowing in what direction • Heatmaps
the target is. • Surveys
Not having a hypothesis (or a flawed • Interviews
one) is one of the most common • Usability tests
reason why A/B tests fail. • Heuristic analysis
(great article here by Peep
Okay, let’s take our initial statement Laja, don’t pay attention to
apart piece by piece to understand the title, it’s great even if
what this means. you do have lots of traffic)

Every test idea should be based on

data […]
Be it quantitative or qualitative, every
test is induced by an analysis of your
website data and the identification of a
problem or element to improve.

Gut feelings or “I read online that…”

won’t do. It could work but you’re A/B
Testing to make decisions based on
data, so let’s not flip a coin for our tests
ideas, shall we?

10 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

[…] an underlying theory […] an informed hypothesis […]
Put yourself in the shoes of your With data and a theory, you can now
customers. Why didn’t they do what craft your test hypothesis.
you wanted them to? What was their
intent at this stage? What would make You can use this format or
YOU leave/not do what was expected? something along those lines:

Example: “I think that if I’m a customer, By {making this change}, {KPI A,

I’d be more inclined to go through with B, and thus primary metric} will
this step if the form had less fields”. improve {slightly / noticeably /
greatly} because {reasons (data,
Don’t skip this. When you’re deep in theory, …)}.
your CRO practice, you sometimes
tend to forget about your users. You Working this way will not only improve
focus on numbers and design. the quality of your tests, but also your
It’s paramount to take a step back. The global conversion optimization efforts.
difference between a bad hypothesis
and a great one could be staring at you Good, you are now testing to confirm
in the face. or disprove a hypothesis, not shooting
from the hip hoping to hit something
Let’s not forget that we are beings eventually.
doted with empathy. We’re humans
creating value for humans.

3. They don’t have a process and a roadmap

1. Why do you need a roadmap?

If you want to succeed What does it consist of?
at A/B Testing, and more
largely at Conversion Rate A roadmap will help you test what
Optimization, you need 2 matters and work toward a clear end
essential elements: goal. It’ll guide your efforts and prevent
you from doing aimless testing.
1. a process, In your roadmap, you should have:
2. a roadmap.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE 11

 • Business goals: the reasons you
have a website. Be concise, simple,
realistic.
• Website goals: how will you achieve
these business goals through your
site. What are your priorities?
• What are your most popular
pages? Which ones have the
highest potential for improvement?
• What does your conversion funnel
look like, step by step? Where are
the friction points?
• Key metrics: how will you measure
success?
• A North Star: What’s the one
metric—correlated with Customer
satisfaction, that if you focus
exclusively your efforts on will
guarantee your success?
(Ex: Facebook=Daily Active
Users, AirBnb= Nights Booked,
Ebay=gross merchandise volume)

Put all these down and make sure

everyone in your team/company
is on the same page and can get
behind them.

12 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

2. Sometimes when we hear But it will look something like that:
“process”, we have an allergic 1. Measure
reaction. We think it means the death 2. Prioritize
of creativity, that it’s boring. You’d feel 3. Test
trapped. 4. Learn
5. Communicate (Don’t skip this.
It’s absolutely not true for A/B Share with your colleagues why you
Testing (or CRO). On the contrary. It’s did that test, how it is pertinent to
BECAUSE you have a process that your business goals, if it succeeded/
you’ll be able to be creative. failed, what you learned. Encourage
discussions, each department has
A/B Testing is a science experiment. insights for different stages of your
You need to be rigorous, have a set of funnel. And keep your IT team in the
parameters making sure what you’re loop.)
doing is measurable, repeatable and 6. Repeat
accurate.

It’s easy to fail. So you need When you feel like you lost your focus,
safeguards. Steps that you will go use Brian Balfour’s question:
through each and every time without
having to think. No need to reinvent
the wheel every time.

You need a process so you can focus

on crafting the best hypothesis
possible and optimize learning. The
details of your process will be specific
to you and your company.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE 13

“
What is the highest-impact thing I can work on
right now, given my limited resources (whether “
that’s people, time or money)?

Brian Balfour, VP of Growth at Hubspot

4. They don’t prioritize

With a process, a roadmap and the For the sake of argument, let’s say they
development of a testing culture inside did have this crazy lift.
your company, you’ll have a list of test
ideas longer than your arm. First thing first:

You need a system to prioritize

tests, to stay on top of things and There are NO colors
make sure you do what matters. And
to not waste time and money on stupid converting more than
tests. others
There are articles on the interwebz
roaming about and claiming things
like: “CHANGING OUR CTA TO RED
INCREASED CONVERSIONS BY 287%”.

14 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

What’s important is visual hierarchy, You can use something like the PIE
i.e. is your CTA standing out from the Framework to rate your tests so you
rest of your page for example. know which one to do first.

Second, testing minute details like

color won’t get you anywhere.

Most often, these type of changes are 1

obvious. If your CTA doesn’t stand out 2
from your page, or if your headline/ 3
copy isn’t clear—do something about
it. You don’t need to test.

If you’d put it in the “Well, duh” Each test will be rated

category, don’t invest your time and following three criteria:
traffic in it. Just do it!
1. Potential gain (../10): How
We’re not saying you shouldn’t test much room for improvement
small things like adding a line of copy is there on this(these) page(s)?
or changing the wording on a CTA. Just 2. Importance (../10): How
don’t make them a priority (except of valuable is the traffic on this
course—as we talked about earlier, (these) page(s)?
if you’re just starting out with A/B 3. Ease of implementation
Testing). (../10): How easy will this test
If your ship was leaking, you wouldn’t be to implement on your site?
put a bandaid on a small hole in the
deck.
Also, keep in mind that testing small
changes or on low traffic pages equals Average the three for each test, and
small results. you’ll have a backlog of ideas ranked
objectively.
It’s true you should test everything,
but you’re working with a finite
amount of ressources.

Once you’re confident in your ability to

A/B Test, focus on tests with the most
impact/potential. Target your pages
with the most traffic and influence.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE 15

5. They don’t optimize for the right KPIs

2. Secondary Actions are not

There are two types of the primary goals of the site, but
conversions. Micro and macro. they are desirable actions that are
You should measure both, indicators of potential future macro
and optimize for the macro conversions.
conversions. Measuring micro conversions
allows you to know where your
friction points are and help you
A micro conversion is a step (click, paint a holistic picture of your entire
social share, newsletter subscription, conversion funnel.
add to cart, …) on the path leading
to a macro conversion, which is an But you shouldn’t optimize for them.
outcome that impacts your bottom- You want to set your test goals as close
line (check out, free trial, …), in other to revenue possible.
words, the main conversion goals of
your website.

Why is it important that you know the

difference?

Because you need to make sure that

you measure both for every test, but You could get more traffic to your
that you don’t optimize for micro landing page through a given variation
conversions only. of your homepage, but have less form
completions even though more people
arrived on it.
There are two types of micro
conversions according to the Nielsen So if you were optimizing only for the
Norman Group: micro conversion of arriving on the
landing page, you would lose money.
1. Process Milestones are con-
versions that represent linear Track micro conversion, but don’t
movement toward a primary macro optimize solely for them. And before
conversion. Monitoring these will starting any test, go back and make
help you define the steps where UX sure you’re measuring everything
improvements are most needed. that matters.

16 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

6. They ignore small lifts

“Go big or go home.” It’s true, as we’ve How’s that for small!
just said, that you should focus your
tests on high impact changes first. Also, the more you test, the more
you’ll improve your website, the less
What you won’t hear us say anywhere you’ll have big results. Don’t be sad
though, is “if your test results in a small if you just get small lifts. It could mean
gain, drop it and move on.” your website is good. It’s pretty rare to
get big lifts on “normal” site.
Why?

Because maths, that’s why.

Don’t be disheartened
Let’s take a quick example.
by small gains, it’ll pay
off big time over time.
If a test gives your variation winning
with 5% more conversions, and each
month you get similar results. That’s a
80% improvement over a year.

80%

J F M A M J J A S O N D

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE 17

7. They’re not testing at all times

Every day spent without an

experiment running is wasted.
Opportunities missed.

Oh, look! An opportuni-aah… It’s gone.

Why are you testing? Because you

want to learn more about your visitors,
make informed decisions, get more
customers and earn more money.

When wouldn’t you want all that?

Never? Yup, same here. So as they say, Next chapter we’ll study the concepts
Always Be Testing. needed to make better decisions as
to when you can stop your A/B Tests
Testing properly takes time, you or not. So you don’t pull the trigger
better not lose any. Test, test, test! too early and end up with fake results
And when in doubt, test again! (yikes).

“
There’s a way to do it better—find it.
“
Thomas Edison

18 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER ONE

 2
YOU’RE
STOPPING YOUR TESTS
TOO EARLY

21
 YOU’RE STOPPING
YOUR TESTS TOO EARLY

This is—without a doubt, the most First problem, most people doing A/B
common, and one of the most potent Testing don’t know—and don’t care,
A/B Testing mistake. whether their tool uses frequentist or
bayesian statistics.
But the answer is really not intuitive.
Second problem, when you dig a bit
We were going to write a chapter into the different solutions, you find
on “when to stop your a/b test for that no 2 softwares use exactly the
both main methods of A/B Testing, same statistical method.
frequentist and Bayesian”.
So, how coud we write something
But, we encountered two problems. helpful?

Here is what we came up with. We will do our best to answer the following
question: What concepts do I need to understand not to stop my A/B Test too
early?

Thus, we will cover:

1. Significance level 3. Duration

2. Sample size 4. Variability of data

Note: None of these elements is a stopping rule on its own, but having a better
grasp of them will allow you to make better decisions.

22 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO

1. Significance level

Don’t trust test results below a 95%

significance level. But don’t stop
a test just because it reached this Statistical significance
level. doesn’t imply statistical
When your A/B Testing tool tells you validity and isn’t a
something along the lines of: “your stopping rule on its
variation has X % chances to beat the
control”, it’s actually giving you the own.
statistical significance level.

Another way to put it is: “there is 5%

(1 in 20) chance that the result you see If you did a fake test, with the same
is completely random”. version of a page, an A/A test, you
have more than 70% chance that your
Or “there is 5% chance that the test will reach 95% significance level at
difference in conversion measured some point.
between your control and variation is
imaginary”.

You want at minimum 95%. Not

less. Yes, 80% chance do sounds like a
solid winner but that’s not why you’re
testing. You don’t want just a “winner”.
You want a statistically valid result.
Your time & money are at stake, so let’s
not gamble!

From experience, it’s not uncommon

for a test to have a clear winner at 80%
significance, and then it actually loses
when you let it run properly.

Okay—so, if my tool tells me that my

variation has 95% chance to beat the
original, I’m golden right?

Well … no.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO 23

2. Sample size

You need a sample representative So, what you need to ask yourself is:
of you overall audience (ignore that
if you want to target a segment in How do I determine if my sample
particular for your test) and large is representative of my entire
enough not to be vulnerable to the audience, in proportions and
natural variability of the data. composition?

When you do A/B Testing you can’t Another issue if your sample is too
measure your “true conversion rate”. small is the impact your outliers will
have on your experiment.
You arbitrarily choose a portion of your
audience with the assumption that
the behavior of the selected visitors
will correlate with what would have The smaller your
happened with your entire audience.
sample is, the higher
You need to really know your audience. the variations between
Conduct a thorough analysis of your
measures will be.
visitors before launching your A/B
Tests.

Here are a couple of examples of

EMAIL
things you need to know:
OTHER
• How much of my visitors come PAID ORGANIC SEARCH
from PPC
• Direct traffic
• Organic traffic
• Email
• Returning visitors
• …

The problem is your traffic keeps DIRECT

evolving, so you can’t know everything
at 100% certainty. REFERRAL

24 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO

What does that mean? Let’s try with
a real-life example. Here’s the data
from tossing a coin 10 times. H (head),
T (tail). We know the “true” probability
of our coin is 50%.

1. We repeat the toss 5 times, and

track the % of heads.

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th %H
T H T H T H T H H T 50
T H T T T H T H T T 30
H T H T T H H H H H 70
H T H T T T T H T H 40
H H H T H T H H H H 80

The outcomes vary from 30% to 80%.

2. Same experience, but we toss the coin 100 times instead of 10.

Tosses %H
100 50 The larger your sample
100 49 size is, the closer your
100 50 result gets to the “true”
100 54
value.
100 47
The outcomes vary from 47% to
54%.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO 25

It’s so much easier to grasp with an But it varies depending on the method
actual example. your tool uses.

With conversion rates, you could have

your variation winning by far the first
day because you had just shot your
newsletter and the majority of your
traffic were your clients for example.

They like you considerably more

than normal visitors, so they reacted
positively to your experiment.

Should you stop the test here, even

with a significance level at 95%, you Here is what we tell our clients that
would have skewed results. use frequentist statistics.

The real result could be the exact Let us insist on the fact that those
opposite for all you know. You made a numbers might not be optimal for you
business decision based on false data. if your tool doesn’t use frequentist
Woops … statistics.

How big should your sample be? That said, we advise our clients to use a
calculator like this one (we have one in
There is no magical number that our solution, but this one is extremely
will solve all your problems, sorry. good too).
It comes down to how much of an
improvement you want to be able to It gives an easy-to-read number,
detect. The bigger of a lift you want to without you having to worry about the
detect, the smaller sample size you’ll math too much. And it prevents you
need. from being tempted to stop your test
prematurely, as you’ll know that till this
And even if you have Google-like traffic, sample size is reached, you shouldn’t
it isn’t a stopping condition on its own. even look at your data.
We’ll see that next.
You’ll need to input the current
One thing is true for all methods conversion rate of your page, the
though: the more data you collect, minimum lift you want to track (i.e.
the more accurate or “trustworthy” what is the minimum improvement
your results will be. you’d be happy with).

26 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO

We then recommend at least 300 earlier. It also could be a bit less if
macro conversions (meaning your there is a considerable difference
primary goal) per variation before even between the conversion rates of your
considering stopping the test. control and variation.
Okay, so if I have lots of traffic and a
I’ll repeat it again, It’s not a magic large enough sample size with 95% in 3
number. days it’s great right?

We sometimes shoot for 1000 Welp, kudos on your traffic, but sorry
conversions if our client’s traffic allows no again …
to. The larger the better, as we saw

3. Duration

You should run your tests for full

weeks at a time and we recommend
you test for at least 2–3 weeks. If
you can, make the duration coincide
with the length of 1 (or 2) business
cycle.
This means, if you started a test on a
Why? Thursday, end it on a Thursday. (We’re
not saying you should test for just one
You already know that for emails and week.)
social media, there are optimal days
(even hours) to post. Test for at least 2-3 weeks. More
would be better though. 1 to 2
People behave differently on given business cycles would be great.
days and are influenced by a number You’ll get people that just heard of
of external events. you, people close to buying while
accounting for most external factors
Well, same thing for your conversion (we’ll talk more about those in our next
rates. Don’t believe me, try. Run a chapter) and sources of traffic.
conversion by day for a week, you’ll see
how much it can vary from one day to If you must extend the duration of
another. your test, extend it by a full week.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO 27

4. Variability of data

fluctuations are negligible considering

the situation and your current rates.
If your significance level, Some tools give you a confidence
the conversion rates of interval, for example “variation A has
a conversion rate of 18,4% ± 1,2% and
your variations are still Variation B 14,7% ± 0,8%”.
fluctuating considerably,
Meaning the conversion rate of the
let your test running. variation A is between (18,4 - 1,2) and
(18,4 + 1,2), and the conversion rate of
the variation B is between (14,7 - 0,8)
and (14,7 + 0,8).
Two phenomenons to consider here:
If the 2 intervals overlap, keep
The novelty effect: When people react
testing. Your confidence intervals
to your change just because it’s new. It
will get more precise as you gather
will fade with time.
more data.
Regression to the mean: This is what
we talked about earlier: the more you
So, whatever you do, don’t report
record data, the more you approach
on a test before it’s actually over. To
the “true value”. This is why your
resist the tentation to stop a test, it’s
tests fluctuate so much at first, you
often best not to peek at the results
have few measures so outliers have a
before the end. If you’re unsure, it’s
considerable impact.
better to let it run a bit longer.

This is also why the significance level

Next up, we dive, as promised into
isn’t enough on its own. During a test,
external validity threats.
you’ll most likely reach several times
95% before you can actually stop your
test.

Wait till your significance curve

flattens out before calling it.
J F M A M J J A S O N D
Same thing with the conversion
rates of your variations, wait till the

28 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER TWO

 3
WARNING!
IS THE WORLD
SABOTAGING YOUR
A/B TESTS?

31
 WARNING! IS THE
WORLD SABOTAGING
YOUR A/B TESTS ?

A/B Testing is great, but let’s be honest Did you raise an eyebrow? Are you
here: if you don’t pay attention, it’s slightly worried? Good. Let’s look into it
pretty easy to fail. together.

You have to create an entire process, What (or who) in the world is screwing
learn how to make hypothesis, analyze up your A/B Tests when you execute
data, know when it’s safe to stop a test them correctly?
or not, understand a bit of statistics …
The World is. Well, actually what are
And… Did you know, that even if you called validity threats. And you can’t
do all of the above perfectly, you ignore them.
might still get imaginary results?

Your A/B Tests results might be flawed if:

1. You don’t send all test data to your 4. You don’t know about the Flicker
analytics tool Effect
2. You don’t make sure your sample 5. You run tests for too long
is representative of your overall 6. You don’t pay attention to real-
traffic world events
3. You run too many tests at the 7. You don’t check browser/device
same time compatibility

32 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE

1. You don’t send all tests data to your Analytics tool

If you see any discrepancies in your

data, or if a metric is not tracked
Always have a correctly, stop everything.
secondary way to Check your goals, metrics, scripts and
start over. Same thing at the end of a
measure and compare test, make sure to double-check your
your KPI’s. results with your analytics tool before
telling anyone.

Cross-check your data like your

business depends on it (and it does).
No tool is perfect, and humans are
even less likely to be. It would be a shame to do everything
right to have flawed results in the end
So, implementation issues with your because your tool was set up wrong or
A/B Testing tool are veeery common. didn’t work properly.

Your goals and metrics might not be

recording properly, your targeted
sample could be badly set up. Just to
mention a couple of possible issues.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE 33

2. You don’t make sure your sample is representative of
your overall traffic

But if that’s not the case, it’s important

you make sure you or your marketing
You can’t measure your don’t have any PPC campaigns,
“true” conversion rate, Newsletters, other marketing
campaigns or anything that could bring
because it’s an ever- unusual traffic to your website during
moving target, new your experiment, thus skewing the
results.
people come and go People coming to your website through
everyday. PPC campaigns will generally have a
conversion rate lower than regular
visitors.

When you do A/B Testing, you take Whereas returning visitors—or worse,
a sample of your overall traffic and your subscribers, will be much more
expose these selected visitors to likely to convert. Because they already
your experiment. know you, trust you and maybe love
you (I hope you have some that do
You then assimilate the results as love you. As Paul Graham, Co-founder
representative of your overall traffic of Y Combinator said: “It’s better to
and the conversion rate measured have 100 people that love you than
sufficiently close to what would the a million people that just sort of like
“true” value be. you”).

This is why you should have traffic One other thing to keep in mind is: if
from all sources. New and returning your competitors are running some
visitors, social, mobile, email, etc in kind of big campaign, it could impact
your sample, mirroring what you your traffic as well.
currently have in your regular traffic.
So pay close attention to your
Unless you’re targeting a specific sample, make sure it’s not polluted
source of traffic for your A/B test that is in any way.
of course.

34 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE

 RETURNING VISITORS

NEW VISITORS

EMAIL

OTHER
PAID ORGANIC SEARCH

DIRECT

REFERRAL

A/B TESTING MISTAKES AND HOW TO AVOID THEM 35

3. You run too many tests at the same time

You’re maybe familiar with Sean Ellis’ and that all your tests actually increase
High Tempo Testing framework. Or you conversions.
just have enough traffic to be able to Let’s not forget that each test has a
run several tests at the same time (well chance to DECREASE conversions. Yup.
done you!). (*shudder*)

BUT—by doing that, you’re increasing You might also be messing up your
the difficulty of the whole process. traffic distribution. My what now?

It takes time to gather data, Let’s take an example

measure, analyze and learn. So if
you run 20 tests in parallel, I hope you Say you want to test steps 1, 2 and 4 of
have an army at your command. Oh, your check out process.

PRODUCT PAGE CHECKOUT PAGE TRAFFIC EXPOSED

A C

XX €

BACK ORDER
XX € BUY

B D

XX €
BUY
XX €

BACK ORDER

Bad traffic distribution

36 A/B TESTING MISTAKES AND HOW TO AVOID THEM

You send 50% of your traffic to your You absolutely can do multiple
control step 1, and 50% to your tests in parallel, but pay attention
variation. So far so good. to traffic distribution, the influence
Except that you’re also testing step 2. tests could have on each other and
that you can indeed handle all these
Meaning that if you don’t re-split evenly tests properly.
(and randomly) people coming out of
both your control and variation from You’re testing to learn, so don’t test for
step 1 so you have an equal amount of the sake of testing. Take full advantage
visitors exposed to your experiment, of the learnings.
your step 2 experiment will be flawed.
And I don’t even talk about the step 4
one.

PRODUCT PAGE CHECKOUT PAGE TRAFFIC EXPOSED

A C

XX €

BACK ORDER
XX € BUY

B D

XX €
BUY
XX €

BACK ORDER

Good traffic distribution

A/B TESTING MISTAKES AND HOW TO AVOID THEM 37

4. You don’t know about the Flicker effect

If it is noticeable, here are possible

reasons:
The Flicker effect
(sounds like a movie 1. Your website is slow to load (it’s bad
both for UX and SEO by the way)
title right?) is when 2. Too many scripts load before your
people catch a glimpse A/B Testing tool’s
3. Something in your test is in conflict
of your control while the with / disables the tool’s script
variation is loading. 4. You didn’t put the script in the
<head> of your page

Optimize for the above reasons, or do

It happens when you use a client-side split URL testing. Just make sure your
tool because of the time needed for control is not noticeable to the human
the JavaScript engine to process the eye.
page. It shouldn’t be noticeable to
the naked eye (i.e. less than 0,0001 If you want to know more about
second). the Flicker Effect and how to fix it,
check out this great article by Alhan
You don’t want people seeing both Keser, Director of Optimization at
versions and wondering what the hell WiderFunnel.
is going on.

38 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE

5. You run tests for too long

“Don’t stop too early, don’t test for too But if you let your test run for too long,
long” … Give me a break, right? your cookies will expire. That’s a real
problem.
Why testing for too long is a problem?
You could also get penalized by Google
Cookies. No, not the chocolate ones if you run a test longer than they
(those are never a problem, hmm expect a classic experiment to last.
cookies).

Cookies are small pieces of data sent

by the websites you visit and stored
in your browser. It allows websites
So make sure the
to track what you’re doing as well as validity period of your
other navigational informations.
cookies is coherent with
A/B Testing tools use cookies to track the length of your test
the experiments.
beforehand.
Cookies have a validity period
(at Kameleoon the duration is
customizable but they last 30 days by
default).
And if you must extend (always by
minimum a full week remember!)
In our case, if you let a test run for
a test because you’re not satisfied
more than 30 days with the default
with your data, don’t forget about
validity period, you’ll have people
them cookies.
exposed to your test several times.
When the cookies expire, your tool
can’t distinguish if the visitor was
already exposed to the experiment or
not, thus polluting the data.

Considering that people sometimes

delete cookies on their own too, you’ll
always have a degree of pollution in
your data. You’ll have to live with that.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE 39

6. You don’t pay attention to real-word events

We already talked about how conversion rates vary between week days. It’s
actually just one case among many other things, outside of your control, that
can influence conversion rates and traffic.

Here are a couple to consider:

Holidays: Pay day: Major news:

Depending on the Yup, it can be a factor in Could be a scandal, as-
nature of your business, some cases. I mean, you sassination, plane crash,
you could either have can probably relate (I and it very well might be
a traffic spike or drop, know I can), when it’s pay impacting your test. If
people having a sense of day, you’re way more people get scared, or if
urgency and converting likely to buy certain the news is sufficiently
faster or the opposite. types of products and important for them to be
make impulse buys. distracted.

Weather: Time of day: You can minimize the

effects by keeping
First the obvious one: if Depending if you’re B2B,
track of the news,
a snow storm is coming or B2C, people won’t
looking at your annual
and you sell parkas, visit and buy at the same
data to look for un-
you’re going to sell times. B2B will usually
usual spikes in traffic,
more as soon as it’s have more conversions
and see if they could
announced. But that’s during business hours,
be caused by external
not all. Weather influ- B2C outside business
events.
ences how we buy. It can hours.
be verified throughout
History, and through
more recent studies like
this one by richrelevance
that found for example
a difference of 10–12%
orders in cloudy vs sunny
days for Clothing, Home/
Furniture, and Wholesale
retailers.

40 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE

7.You don’t test for browser / device compatibility

Be particularly careful with this if you

use your A/B testing tool visual editor
This one is a make to make changes more complicated
or break. Your test than changing colors or copy (we
advise you avoid doing that, and code
should work across all yourself for this types of changes). The
browsers & devices. generated code is usually ugly and
could be messing things up.

Test your code on all browsers and

If variations don’t work or display devices. Chrome inspecting tool is
properly depending on device and usually good enough to simulate all
browser, you’ll end up with false devices (or through your a/b testing
results (as malfunctions will be preview feature, if it has one like ours).
counted as losses). Not to mention Don’t use browser-specific CSS or Ajax.
awful user experience. Be careful with too recent CSS (check in
your analytics tool what browsers your
Don’t forget that 80% of internet users audience is using).
own a smartphone.
Next, we’ll make sure you’re not misin-
terpreting your results.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER THREE 41

 4
ARE YOU
MISINTERPRETING
YOUR TESTS RESULTS?

43
 ARE YOU
MISINTERPRETING YOUR
TESTS RESULTS?

Once you have a process, know how A/B Testing is about learning, and
to formulate a hypothesis, set up your making informed decisions based on
A/B tests and know when to press stop, your results.
you’re all good, right? You’re on top of
your A/B Testing! So let’s make sure we don’t mess that
up!
Nope.

Making sure you’re correctly

interpreting your results is just as
important.

In this chapter, we’ll look into ways you’re possibly misinterpreting your
tests results:

1. You don’t know about false 3. You’re testing too many variables
positives at once
2. You’re not checking your segments 4. You’re giving up on a test after it
fails the first time

44 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR

1. You don’t know about false positives

Are you aware that there are actually 4 “Yes, it’s true that a team at Google
outcomes to an A/B Test? couldn’t decide between two blues, so
they’re testing 41 shades between each
What do you mean, it’s either a win or blue to see which one performs better.
a loss, no? I had a recent debate over whether a
border should be 3, 4, or 5 pixels wide,
Nope, it can be: and was asked to prove my case. I can’t
operate in an environment like that. I’ve
• False positive (you detect a grown tired of debating such minuscule
winner when there are none) design decisions...”
• False negative (you don’t detect a (You can read the full article here).
winner when there is one)
• No difference between A & B Whether you agree or not with
(inconclusive) him that it’s wrong from a design
• Win (either A or B converts more) standpoint, it’s also mathematically
wrong depending on how you do it.
(If you’re a bit hardcore and want
to know more about this, check out
hypothesis testing. It’s the actual
mathematical method used for You have two ways of ap-
(frequentist) A/B Testing.) proaching this:

Why should you care? Because you 1. You do “cascade testing”,

could have been interpreting false i.e. A vs B, then B vs C, then C
positives as genuine wins. And in- vs D, … THIS IS BAD, DON’T DO
vested money in it. IT. We’ll see why in a second.
2. You do A/B/n testing,
Take the well-known example from meaning you test all variations
Google: 41 shades of blue. (Nope, it in parallel.
has nothing to do with the books. What
books? No idea, you?)

Doug Bowman, Lead designer at

the time, actually left (but for design
reasons) the company because of this:

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR 45

1. Cascade Testing 2. A/B/n Testing
Imagine you want to test a different A/B/n Testing is when you test n
headline for a product page. You have number of variations instead of
your current one (A) against the new just one (B) against your control (A).
(B). B wins, but your boss doesn’t Meaning you have your control A,
like the wording and wants you to against variation B, C, D, E, F, etc. at the
try a slightly different wording. Then same time, in the same conditions.
you feel like you could do better and
change it again. And again. This is absolutely fine. BUT, as we
saw in our chapter on when to stop
You end up testing 10 different your A/B tests, you need at least 300
variations of this headline. How is it a conversions PER variation to call the
problem? test off if you’re using a frequentist tool
for example.
Let’s take a look: A vs B gave B as the
winner with 95% statistical significance. In our Google example, you would
As we saw in a previous chapter, it need 41x300 = 12300 conversions.
means that there is a 5% chance this That’s a lot.
result is a complete fluke or a “false
positive”. If you have Google-like traffic it’s okay.
If you’re like us mere mortals though,
Then you tested a third headline, B vs this is a big fat loss of time.
C. C also won with 95% significance.
The problem is that the chance of a You could even be testing for too long
false positive compounds with the and get skewed results. This kind of
previous test. Your second test winner, tests is very rarely needed and can
C, has actually 9% chance of being a often be completely avoided by
false positive. having a better hypothesis.

After 10 tests on your headline (C vs D,

D vs E, …), even with 95% significance
on your tenth test, you actually have
a 40% chance of your winner being
a false positive! (For 41 variations it
becomes 88%!!!)

You’d be flipping a coin. Or deliberately

shooting yourself in the foot depending
how many times you repeat this.

Don’t do cascade testing. Just don’t.

Okay? Kittens will die if you do.

46 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR

 Cascade Testing

A VS B
5%
chance of being
1 a false positive

B VS C
9%
1

C VS D 40%
(...)

K VS (...)

A/B/n Testing

A VS B VS C VS D (...) K VS (...)

5%
chance of being
a false positive

A/B TESTING MISTAKES AND HOW TO AVOID THEM 47

2. You’re not checking your segments

Don’t make Avinash Kaushik sad (one of Web Analytics’ daddies if you’re
wondering).

He has a rule:

“ Never report a metric without segmenting it to

give deep insights into what that metric is really
hiding behind it. “
Most data you get from your An experiment could be resulting
analytics tool is aggregated data. It in your variation losing overall but
takes all traffic and mashes it out into winning on a particular segment.
pretty but absolutely not actionable
graphics. Be sure to check your segments
before closing the book on a test!
Your website has a number of func-
tions, your visitors come with different
objectives in mind. And even when
they come for the same reason, they
probably don’t need the same content.
Important side note: when checking
If you want an effective website, segments in an experiment’s results, be
you can’t consider your traffic as a sure not to forget that the same rules
faceless blob, you need to segment. apply concerning statistical validity.
Before declaring that your variation won
It also applies for your tests results. If on a particular segment, check you have
you don’t segment them, you could be enough conversions and a large enough
wrongly dismissing tests. sample size on that segment.

48 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR

Here are three ways you can segment your data:

1. By source 2. By behavior 3. By outcome

Where do your visitors What do they do on Segment by the ac-
come from (Ads, Social your website? People tions people took on
Networks, Search behave differently your website: bought a
Engines, Newsletter, …)? depending on their product, subscribed to a
Then you can look at intent / needs. newsletter, downloaded
things like: what pages You can ask: what con- a premium ressource,
they go depending tent do people visiting applied for a loyalty
on where they come your website 10+ times card, …
from, their bounce rate, a month read vs those
difference in loyalty, if only coming twice? What Make groups of visitors
they come back… page people looking with similar outcomes
at 5+ pages on a visit and ask the same type
arrived on vs people of questions we asked
who just looked at one? above.
Do they look at the same
products / price range? You’ll see what
campaigns worked, what
products to kill, etc…

By segmenting you get actionable data and accurate

results. With actionable data and accurate results
you can make informed decisions, and with informed
decisions … $$$$!

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR 49

3. You’re testing too many variables at once

You got the message, you need to test What if one of those positively
high-impact changes. impacted conversions and the others
dragged it down? You counted the test
So you change the CTA, headline, add as a failure and it wasn’t one.
a video, a testimonial and change
the text. Then you test it against your Make sure to clearly specify what
current page. And it wins. success looks like and that you’re set
up to measure it.
Good, right?

Well… Not really. If you can’t measure it, you won’t

learn. If you don’t learn, you can’t
How will you know which one(s) of reproduce nor improve it.
your changes improved conversions
on your page vs dragged it down? Don’t test multiple variables at
once. Unless you know how to do
This is where the question “How will I multivariate testing, then it’s fine. But
measure success” takes all its meaning. as it requires a gigantic amount of
Testing is awesome, but if you can’t traffic, we rarely see it used.
really measure what happened, what
moved the needle, it’s not so useful.

What have you learned? That some

combination of your changes improved
conversions?

50 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR

 Don’t test too many variables at once

Don’t test too many variables at once

CONTROL VARIATION

CONTROL VARIATION
VS

VS

How would you know which changes IMPROVED vs DECREASED conversions?

How would you know which changes IMPROVED vs DECREASED conversions?

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR 51

4. You give up on a test after it fails the first time

If you followed our guidelines on how mark in the “too bad” column, conclude
to craft an informed hypothesis, each clarity wasn’t in fact an issue and move
of your tests should be derived from on to another test?
(best should be a combination of
several): No, of course you don’t. A/B Testing
is an iterative process.
• Web Analytics
• Heatmaps Take another look at your data, devise
• Usability tests ways to improve your page.
• User interview
• Heuristic analysis • You could add testimonials
• You could remove information not
For example, you have analytics data relevant to the product
showing people staying a while on your • You could add a video
product page, then leaving. • …

You also have an on-page survey As you now know not to do cascade
where visitors told you they weren’t testing (which is completely different
quite convinced that your product than iterative because you don’t test X
answered their need. versions of the same headline/picture
against the winner of a previous test),
Your heuristic analysis showed you had or test everything at once, you can
clarity issues. embrace iterative testing.

Click maps show people going through

all your product pictures.
There isn’t just ONE
You then decide to test changing the
copy and add better pictures on the
solution to a given
page to make it more clear. problem. There are an
Your test ends and … results
infinite number of them,
are inconclusive, no increase in and it could very well
conversions.
be a combination of
What do you do now? You put a check several solutions.

52 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR

“ If you’re not stubborn, you’ll give up on experiments
too soon. And if you’re not flexible, you’ll pound
your head against the wall and you won’t see a
different solution to a problem you’re trying to
“
solve.

Jeff Bezos

Let’s be a tad extreme to illustrate this. Same thing with your A/B tests!

When your internet cuts off, what do Don’t give up or jump to conclusion
you do? If you’re plugged through as soon as something doesn’t work.
an ethernet cable, maybe you try Look for other solutions and test
unplugging/re-plugging it. again, and again.

If it doesn’t change anything, do you Okay, you are now aware of

then conclude that your cable is dead, several ways you could have been
and go buy a new one? misinterpreting your A/B tests results,
we’re making progress!
Or rather you try to plug it in an
another computer, go check your For our last chapter, we’ll take a look
router, restart your computer, check inside our brains, how they could be
your drivers, … playing tricks on us and jeopardizing
our A/B Tests *cue spooky music*.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FOUR 53

 5
YOUR BRAIN
IS YOUR WORST ENEMY
WHEN A/B TESTING

55
 YOUR BRAIN
IS YOUR WORST ENEMY
WHEN A/B TESTING

Did you know that we, humans, SUCK Cognitive biases are then a real threat.
at statistical reasoning?
To get that out of the way, cognitive
We’re also irrational, flawed, and sub- biases are personal opinions, beliefs,
jective. preferences that influence your ability
to reason, remember, evaluating infor-
Why? Because we’re influenced by a mation.
list of cognitive biases longer than your
arm. Let’s go down the rabbit-brain (sorry,
had to do it) and make sure we’re not
You can perfectly live (but biased) subjectively influencing our tests too
without knowing about them, but much by being our flawed (but lovable)
if you’re here, it means you’re A/B selves.
Testing or contemplating to start so.

A/B Testing is a science experiment

which must by definition be objective
to provide actionable data.

56 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE

Here’s what your brain does and what we’ll cover:

1. Will find a relationship between 6. Will base all subsequent thinking

unrelated events on the first piece of information
2. Can’t handle sample size received
3. Will subconsciously look for and 7. Will throw reason out the window
interpret information to confirm its as soon as ego/emotion is involved
own beliefs 8. Prevents you from thinking like
4. Will see patterns when there are your customers
none and thinking that past events 9. Takes things for granted because
influence future probabilities you’ve always done them this way
5. Thinks what’s in front of him is 10. Overestimates the degree at
everything he needs to draw which people agree with you
conclusions

1. Finding relations between unrelated events

Remember how we talked about Why an A/B Test worked or not isn’t
external validity threats? straightforward. Be careful not to
rush your test analysis.
Well, if you didn’t know about them,
you could assume that the lift you see Our brain jumps to conclusions like
was indeed caused by the pink CTA there is no tomorrow. (A great book on
you put in your variation. Not because the subject is “Thinking Fast and Slow”,
there is a storm coming that scared by Daniel Kahneman.)
people into buying your product for
example.

You’d have been victim of the illusory Your results are what
correlation bias. You perceived a rela-
tionship between 2 unrelated events.
you’ll use to take
business decisions, so
don’t rush your analysis.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 57

2. Can’t handle sample size

When we talked about fixing a sample size before testing, we actually were also
partially preventing another bias called insensitivity to sample size.

Our brain struggles to apprehend correctly sample size and underestimates

variations in small samples.

Example from D. Kahneman’s book:

A certain town is served by two hospitals. In the larger hospital about 45 babies are
born each day, and in the smaller hospital about 15 babies are born each day. As
you know, about 50% of all babies are boys. However, the exact percentage varies
from day to day. Sometimes it may be higher than 50%, sometimes lower.

For a period of 1 year, each hospital recorded the days on which more than 60% of
the babies born were boys. Which hospital do you think recorded more such days?

1. The larger hospital 2. The smaller hospital 3. About the same

(that is, within 5% of each other)

Care to make a guess?

Here’s what the answers looked like in the study:

“56% of subjects chose option 3, and 22% of subjects respectively chose options 1
or 2.

58 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE

However, according to sampling theory Don’t draw any conclusions from small
the larger hospital is much more likely samples as results won’t have any
to report a sex ratio close to 50% on statistical value. They can spark ideas,
a given day than the smaller hospital discussions and be the basis for actual
which requires that the correct answer tests, so you can verify the data.
to the question is the smaller hospital.”
You’re warned of the risks now, so
Sample size is capital, and our think hard before making a decision
brain usually forgets about it when based of a small sample.
considering problems similar to the
example above.

3. Looking for and Interpreting information to confirm

your own beliefs

Confirmation bias is also not to be • Does it really prove your

ignored. It’s the fact that you will seek, hypothesis in an objective way?
interpret or focus on information (sub- • Did you push this idea with your
consciously or not) that confirm your ego on the line to begin with?
beliefs. Add to that the Congruence • Aren’t there other factors that
bias, where you test what YOU think is could have produced (or at least
the problem, rather than the rest and considerably helped) this lift?
you got yourself a nice, subjective test.
For example, if you think the color red
does indeed increase conversions, If you’re testing to prove you’re right
your brain will look for any information and/or you value ideas themselves,
to confirm this belief. you’re doing it wrong.

A test could barely go in your direction

(not in a statistically significant way),
you’ll be way more inclined to call it a You’re testing to learn,
success than if your convictions weren’t
on the line.
not building your
ego. Impact is what’s
Every time you feel you were right, and
that data seem to go your way, it’s time
important.
to pause and ask yourself:

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 59

4. Seeing patterns when there are none and thinking
that past events influence future probabilities

Let’s tackle two biases at once.

The clustering illusion: the intuition Okay, let’s keep flipping coins. Let’s say
that random events which occur in we flip another coin 39 times, and get
clusters are not really random events. 39 heads in a row.

A fun story illustrating the clustering What is the probability of having heads
illusion is the one about the Texas again for the 40th flip?
Sharpshooter.
50%. Just as any other coin flipping.
It’s the story of a Texan who shoots on
the blank wall of his barn then draws If you were a bit confused, you fell prey
a target centered where his shots are to the gambler (or hot hand) fallacy.
most clustered. And then he proceeds
to brag about his shooting skills. You thought that because you got
heads so many times in a row it would
Because you see similarities doesn’t somehow influence the probability of
mean there is a pattern. Nor because the last throw.
you made some good guesses in the
past mean you’ll keep making them.
Flipping a coin 10 times and getting
7 tails doesn’t necessarily means the
coin is biased. It just means you got
tails 7 time in a row

60 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE

Don’t stop a test because “you see
a pattern” or “spot a trend”. Think
about those 2 biases, odds are over- Only data, obtained
whelmingly in the favor of what you through rigorous tests,
think you see is random.
will tell.
Maybe you got some good results
based on your intuition. Or so you
think. You could actually be in the
same position as our Texan shooter.

And because you’ve been right twice

about something before, doesn’t
mean you will be next time.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 61

5. Thinking what’s in front of him is everything he needs
to draw conclusions

This is what D. Kahneman called “what you see is all there is” in his book. It’s the
notion that we draw conclusions based on information available to us, i.e. in front
of our eyes.

Doesn’t sound too bad, uh?

Let’s try with this:

+ = 1€10

A bat and a ball together cost $1.10. Your brain is wired to look for patterns,
The bat costs $1.00 more than the ball. and drawing conclusions with what you
How much does the ball cost? have. Except he sometimes jumps the
gun.
50% of the students who were
asked this simple question, students Because you’ve got 2 pieces of data
attending either Harvard or Yale, got under your nose, doesn’t mean
this wrong. they’re all you need to draw a sensi-
ble conclusion.
80% of the students who were asked
this question from other universities
got it wrong.

I’ll let you find out the answer on your

own. And no it doesn’t cost 0.10$.

62 A/B TESTING MISTAKES AND HOW TO AVOID THEM

6. Basing all subsequent thinking on the first piece of
information received

Called the Anchoring bias, it’s the When the number given is precise,
fact that we allocate more importance people tend to negotiate in smaller
to the first piece of information we’re increments than with round numbers
given. (studies on the topic here).

Here is an example from a study by If the interviewer went first, with his
Fritz Strack and Thomas Mussweiler: highest bid—or what he said was his
2 groups of people were asked about highest bid, I’d wager you used it as a
Gandhi’s age when he died. base with your counter-offer instead of
what you thought you were worth.
The first group was asked if it was
before 9 years old or after. The second By now you must be getting weirdly
if it was before 140 or after. Both suspicious of your own brain. Good.
answers are pretty obvious. Being aware that we’re built to jump
to conclusions, consider only what’s
But what was very interesting, were the in front of our eyes and ignore the
answers from both groups when they big picture is the first step in the right
were asked to guess Gandhi’s actual direction.
age when he died.
Be extra-careful with your numbers
Answers from the first group had an and tests! When you feel you’re sure
average of 50 vs 67 for the second. about a result, pause and check
Why such a difference? again. Run a the test a second time
Because they were subconsciously if needed.
influenced by their respective first
questions.

Think about your last salary

negotiation for a job interview. The
first person to give a number basically
calibrates the rest of the negotiation,
because everything following will be
based on that number, for the starting
value and the scale of the negotiation.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 63

7. Throwing reason out the window when ego and emo-
tion are involved

This one can be painful. Swallow your frustration, go back to

the hypothesis that led to this redesign,
You put your heart and soul in a aaand to the drawing board again.
redesign, you spent hours on it and
you’re super-proud of what you made.

Then you test it. And it flops. Badly. Being able to throw
Ouch …
out hours —days even,
What do you do? of work through the
window if your data
“Screw these people my design is
perfect, they don’t know what they’re says so, is a sign you’re
talking about!”
truly becoming data-
Or when you bring the news to your driven.
boss he says: “No way, this design is
clearly better, go with this one”

No! Be strong, I know your pain. It’s

It’s freakishly hard though.
hard but that’s one of the reasons
you’re A/B Testing, to not lose
money on redesigns or decisions
based on guts and personal
opinions. But rather do things people
actually want.

64 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE

8. Preventing you from thinking like your customers

Called the curse of knowledge, it’s Don’t ask someone from your team
when you’ve been so absorbed by a though. You could all be victims of
subject that you’re having a hard time the bandwagon effect. Members of
thinking about problems like someone a group influence each other. And the
who has little to no knowledge about it. more people do something, the more
other people might be influenced to do
When you know something is there— the same.
say a new button or a new picture,
that’s all you see on the page. If you don’t regularly receive
external feedback, you might have
But your visitors could just as well not built yourself a distorted reality.
even see the difference.

Ask someone else, a colleague from

another team to take a look. Or you
could do usability tests.

Regularly ask your

visitors and clients, as
well as other teams for
feedback.

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 65

9. Taking things for Other things you can do
to try and trigger lateral
granted because you’ve
thinking:
always done them this
way • Turn your problem on
its head, try to solve the
opposite
Functional fixedness is when
you’re stuck in linear thinking. • Think about the most obvi-
ous, stupid solution
It means that because you’ve • Fragment your problem
always done something in a certain in series of small, very
way or used a tool a certain way, precise problems
you don’t question it and go with it.
• Don’t be satisfied with
You see an iron, you think about finding one solution
its obvious use—for clothes, you • Flip your perspective, how
don’t think to use it as a toaster if would you think about
you don’t have an oven (or a real this problem if you were
toaster). an engineer, a scientist, a
complete beginner?
This is called lateral thinking, or
“thinking out of the box”. Easier
said than done though, uh.

You can try to trigger this type of Your Thinking

thinking by repeatedly asking “why“
every time you think something
is obvious. That way you’ll dive in
your assumption till you get to its
bottom—or source.

What you’ll find might blow mind,

sideways.

66
10. Overestimating the degree at which people agree
with you

“Everyone hates popups.” When you think that something is

doing as good as it could be, stop in
Well, YOU hate them. They usually your tracks and reconsider.
annoy people a bit but they actually
convert quite nicely when used the
right way (i.e they don’t pop in your
face the second you arrive on a
website).
Test what would have
the most impact, but
This type of bias (aka the false
consensus effect) can be tricky when
also test what is doing
gathering feedback. fine. You can always do
We can sometimes believe strongly
better.
in something, and think we’re on the
side of majority when we’re really
not. We tend to assume people have
the same opinions we do. Particularly
in a small group, when we arrive to a
consensus on something, we’re very
quick to think it’s representative of the
overall population.

That’s why it’s better to get

feedback from individuals rather
than from a group. Group feedback
will be plagued by biases.

But when asking individuals, don’t get

caught up in personal preferences
either. Always take everything with a
grain of salt.

Be careful not to do it yourself too!

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 67

That’s all on cognitive biases… and for Back to you now,
our series on A/B Testing Mistakes.
let’s go get some
Congratulations! You went through all testing done, mistake
67 pages of this ebook! We hope you
got some useful nuggets , learned a bit free!
and that your tests will get better as a
result.

68 A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE

 NEVER MISS
OUR EPIC CONTENT

If you’re serious about optimizing your website,

we bet you’ll love our monthly newsletter.

More ebooks like this one will come too.

SIGN UP FOR FREE

A/B TESTING MISTAKES AND HOW TO AVOID THEM CHAPTER FIVE 69

 Kameleoon is the most advanced optimization platform on
the market. Simple, fast and reliable, our SaaS A/B testing and
personalization solutions have been designed to allow marketing
teams to craft optimized and personalized experiences to each
of their visitors and to take the right decisions at the right time.

Kameleoon’s solutions rely on an architecture that treats real-

time visitors data and runs predictive targeting algorithms able
to identify optimal segments for personalization campaigns.

Several hundreds of clients, from pure players to large firms,

have successfully adopted Kameleoon’s solutions to boost their
conversions.

FOLLOW US © Kameleoon, 2016 www.kameleoon.com

FRANCE GERMANY RUSSIA

12, rue de la Chaussée d’Antin Beim Alten Ausbesserungswerk 4 105187 Россия, Москва,
75009 Paris, France 77654 Offenburg, Deutschland ул. Щербаковская, 53/3
+33 (0) 1 83 62 20 50 +49 (0) 781/924 17-726 +7 910 413 20 83
info@kameleoon.com info@kameleoon.de info@kameleoon.com