Esomar 28

28 questions to help research buyers of online sample

Last reviewed: January 2015

Company Profile
Q1. What experience
does your company have
with providing online
samples for market research?

Context: This answer

might help you to form
an opinion about the
relevant
experience of the sample
provider. How long has

the sample provider been

providing this service and do they have for
example a market research, direct
marketing or more technological
background? Are the samples solely
provided for third party research, or does the
company also conduct proprietary work using their
panels?

Toluna is one of the worlds leading online panel

and survey-technology providers. With 18 offices
in Europe, North America and Asia Pacific, we
work with many of the worlds leading
market-research agencies, media agencies, and
corporations. Through the years, Toluna has built
and nurtured one of the largest and most diverse
qualified online panels in the world. Our
proprietary panel-recruitment methods, profiling
technologies and sample management techniques
yield access to more than 7.8 million respondents
from 46 countries. We also offer as many as 750
data points on each panelist, thereby enabling
companies to interview many reach hard-to-reach
audiences with ease and efficiency.
Key acquisitions:
In 2008, Toluna welcomed Common Knowledge
Research Services, and their Your2Cents Online
Opinion Panel, into the group, complementing our
traditional strengths in Europe and Asia Pacific with
solid online sample capability in the US and
Canada. In 2009, Toluna acquired the survey
services division of Greenfield Online/Ciao Surveys,
expanding Tolunas global respondent reach, and
footprint in the US market.

Industry associations
and guidelines followed:
Toluna adheres to the
globes most stringent
guidelines for panel
management and
data quality. We are
members of many
industry associations committed to innovating
data quality. Toluna is a licensed TrueSample
sample provider. The company has several
executive members of ESOMAR and participates
in its Professional Standards Committee which
defines and reviews the codes and guidelines of
the market research industry. Our executive staff is
also member of the Market Research Association
(MRA), American Marketing Association (AMA),
the Canadian Market Research and Intelligence
Association (MRIA), the German Society for Online
Research (DGOF), and the German Association for
Market and Social Research (BVM). Toluna also
has corporate delegates on The ARF Foundations
of Quality Board (FoQ 2), the CASRO Board,
and the Syntec Etudes Board (French research
association). Toluna is Greenbook Healthcertified.

Sample Sources & Recruitment

Q2. Please describe
and explain the type(s)
of online sample sources
from which you get
respondents. Are these
databases? Actively
managed research
panels? Direct
marketing lists?
Social networks? Web
intercept (also known as
river) samples?

Context: The
description of the types
of sources a provider
uses for delivering an
online sample will
provide insight into the
quality of the sample.
Toluna offers access to
46 actively managed
proprietary panels
around the world.
We recruit members
using various methods
including web-banners,

website referrals, pay-per-click, natural search

optimization, affiliate marketing, email, and online
public relations activities.

Respondent Detection
technology, email
address, and more.

Toluna takes cues from social media and new

technology to provide respondents with a relevant
and rewarding experience. By enhancing the
experience for Toluna.coms diverse members, we
build loyalty, trust and engagement. This yields
accurate and deep member profiles, and improves
respondent targeting as well as data quality.
Community features include;

Countries where RTS is

used (usage varies by
country);

Personal member profile pages that provide

information about rewards, and incentives
earned
The ability to post and respond to on-site
comments from other members
The ability to create on-site polls, and vote
The ability to upload images, video and more
A rich member App that enables members to
engage with Toluna.com, post polls, thumb its
and battles on Toluna.com via mobile devices
The ability to engage with and follow others
In addition, Toluna has developed an approach to
fulfill the sample requirements of researchers that
differs fundamentally from existing, conventional
online panel methods. This approach, referred to
previously as river samples, is known at Toluna
as Real-Time Sampling (RTS). It consists of
recruiting individuals in real- time from a network of
websites with which Toluna has developed referral
relationships. This methodology taps into the
many potential survey takers online who are willing
to participate in surveys, but may not necessarily
want to join a market research panel.
Potential survey respondents are then asked a
series of demographic (and attitudinal) questions.
They are then directed to available surveys for
which they might qualify, and for which clients have
approved the RTS approach.
Toluna has the ability to track future visits from
these survey takers using Duplicate

Australia
Austria
Belgium
Brazil
Canada
China
Denmark
Finland
France
Germany
Greece
Hong Kong
India
Ireland
Italy
Japan
Malaysia
Mexico
Q3. If you provide samples from more than one
source: How are the
different sample sources
blended
together to ensure validity? How can this be
replicated over time to
provide
reliability? How do you
deal with the
possibility of
duplication of
respondents across
sources?

Netherlands
New Zealand
Norway
Poland
Portugal
Russian Federation
Spain
Sweden
Switzerland
Thailand
Turkey
United Kingdom
United States

Context: The variation in

data coming from different sources has been
well
documented.
Overlap between
different panel
providers can be significant in some cases and
de-duplication removes
this source of error, and
frustration for respondents.

At Toluna, we refer to our process for ensuring

data quality as SmartSampleTM. SmartSample
ensures that respondents are (a) real, (b) valid, (c)
unique, (d) engaged, and (e) representatiive, as
described below;

A) Real Respondents are GeoIP and postal-code

validated. using Postal Address File (PAF) (Melissa
data in the US). Imperiums Address
Correction product is used to prevent individuals
from participating in surveys unless a valid postal
address is used. In addition, each respondent
completes a CAPTCHA confirmation process.
Additionally, Toluna panelists complete Tolunas
double-opt-in registration:
Individuals must confirm membership by
clicking on an emailed link upon completion of
the Toluna enrollment survey.
Full membership is achieved only after
completion of one professional survey.
B) Valid Toluna blocks individuals with known
fake or disposable domains. When possible,
respondents are further validated using the TrueSample validation process, which compares an
individuals name and address with third-party
sources, including Postal Address File (PAF) (Melissa data in the US), telephone directories, etc.
TrueSample validation is available for respondents
in the U.S., Canada, UK, Germany, Spain & Italy.
For our proprietary healthcare professional panels
in the UK, France, Germany, Italy and Spain, we
collect web urls and phone numbers for the physicians practice to validate. Our US physicians
are AMA (American Medical Association) member
validated when they first join, and again each time
they attempt to take a survey.
C) Unique - Respondents cannot enroll in the
Toluna community, or participate in the same
survey more than once, fraudulently or accidentally.
Tolunas Duplicate Respondent Detection
cookie-based duplication-detection technology is
used during the panelist-registration process, and
at the beginning of every Toluna survey. This is
coupled with a third-party digital fingerprinting
technology, Imperiums RelevantID, to prevent
respondents from participating in surveys more
than once. As a further measure to prevent
duplication, Toluna uses technology to produce a
proprietary, matching algorithm to flag similarities of

new registrants with

existing Toluna
panelists.
D) Engaged The
following are some of
the checks utilized to
ensure respondent
engagement:
Speed-checks are applied to any user who
completes a survey in less than 1/2 of the
median survey duration.
Straight-lining checks can be applied to any
grid question with at least 4 scale levels and
15+ items (Toluna validates with clients the
appropriateness of these checks on a
question-by-question basis).
Red herrings (trap questions) are used in most
questionnaires to validate the respondent
attention.
In addition, we can accomodate other trap
questions that compare prior responses within
survey, or demographic questions against profile
data.
E) Representative Representativeness is achieved
by selecting a sample that reflects the population-of-interest accurately. This can be achieved
through the appropriate application of simple or
stratified random sampling, quota
sampling, or by using more advanced techniques
such as SmartSelect, an individual respondent
selection method that ensures that the
characteristics of each respondent for each
project, irrespective of originating sample source,
match those of the target population of interest.
At Toluna we offer all of these techniques and test
them to ensure the results are representative of the
population-of-interest. In addition to using
appropriate sampling techniques,
representativeness can be achieved by weighting
the data to be representative of the population-ofinterest. Toluna experts understand the
appropriate weight function to apply and
appropriate variables to use in the weighting
algorithm.

Q4. Are your sample

Context: Combining
source(s) used solely for
respondents from
market research? If not,
sources set up
what other
primarily for different
purposes are they used
purposes (like direct
for?
marketing for
example) may cause undesirable survey
effects.
Yes, Tolunas panel communities are accessed
solely for the purpose of market research. Under
no circumstances do we send any form of either
advertising or direct marketing to our members.
Context: Ensuring the in- Q5. How do you source
clusion of hard-to-reach
groups that may be
hard-to-reach on the
groups on the internet
internet?
(like ethnic minority
groups, young people,
seniors etc.) may increase population coverage and
improve the quality of the sample provided.
Tolunas community panels, and the broad range of
polls and opinions that they create, attract a wide
variety of respondent groups, profiles and
demographic clusters.

array of media and affiliate programs to recruit

on demand specific
socio-demographic
segments for our global
panel community
network, and our team
is highly experienced in
tailoring communications for a multitude of diverse
audiences - with surveys completed across 73
countries.
An optimized and carefully planned registration
process ensures maximum conversion to
membership while a robust profile management
process ensures a seamless and gradual
completion of additional profiling data identifying
those hard-to-reach respondents.
Toluna is uniquely equipped to provide reliable
access to broad consumer targets, as well as,
extremely niche profiles because of these
sophisticated recruitment and profiling tools. In
cases where we need to reach hard-to-reach
respondents we may also consider using Tolunas
QuickVotes, and profile Toluna.com community
members in real-time, for future surveys.

Together with the ability to attract panel community

membership from niche audiences using media
placement, and partnerships with specialist media
titles, Toluna has a unique capability to find and
survey hard-to-reach audiences. In addition, we
strongly leverage social and mobile platforms to
attract people to Toluna.com.
The user-driven content and interactive value of
Tolunas member community, Toluna.com
has broad appeal, and has doubled visitor
conversion via word-of-mouth/referral and/or
natural search optimization, while attracting very
broad and/or traditionally very hard to reach
demographic profiles of consumers (such as teens)
as well as consumers not normally pre-disposed to
join an online panel.
Tolunas dedicated panel managers utilize a broad

Q6. If, on a
particular project, you
need to
supplement your
sample(s) with sample(s)
from other providers,
how do you select those
partners? Is it your
policy to notify a
client in advance when
using a third party provider?

Context: Many
providers work with third
parties. This means that
the quality of the sample
is also dependent on
the quality of sample
providers that the buyer
did not select.
Transparency is
essential in this
situation. Overlap
between different
providers can be
significant in some cases and de-duplication
removes this source of error, and frustration for
respondents.
Providers who observe process standards like the
ISO standards are required to give you this information.
The Toluna Affiliate Network is used in cases where
we need to augment our proprietary panels, and
Real-Time Sampling capability.
Toluna only works with respondent providers who
have their own proprietary panels we do not
work with brokers. This means we work with local
suppliers who are typically based in the country
under study. We evaluate and select suppliers
who have local knowledge of the regions where we
are carrying out studies.
Suppliers follow all industry guidelines, provide
adequate responses to ESOMARs 28 questions,
and partners are asked for opt-in policies, privacy
policies, legal guidelines, and geographic locations.
Once a supplier is on-board, it is automatically and
continuously monitored to gauge performance. If
an agency fails to deliver against commitments
on two studies, its status is set to probationary
until the issues are resolved and performance is
restored. If the supplier continues to fail to deliver
while on probation, then it will be removed from our
preferred supplier list.

Performance is
measured on several
factors:

Speed of response
Level of profiling
Cost
Ability to deliver
Speed and
efficiency

Toluna has a minimum of two to three suppliers in

each country, so that we have backup in case one
supplier fails to deliver. Clients are informed when
a partner is required.

Sampling & Project Management

Q7. What steps do you
take to achieve a representative sample of the
target population?

Context: The
sampling processes (i.e.
how individuals are
selected or allocated
from the sample sources)
used are the main factor
in sample provision. A systematic approach based
on market research fundamentals may increase
sample quality.
We take pride in our research sampling expertise,
which requires investment not only in processes
and systems, but in the skills of the team
responsible for sampling.
For our panel, sample selection is made randomly
using the profile criteria specified in client quotas,
taking account of predicted response rates by
target demographic and country to avoid
over-contacting panelists and to ensure that we
do not introduce a bias in the responses. Historical
propensity to answer surveys is not used to select
a sample. Respondents are randomly selected for
surveys that they have a likelihood of qualifying for.
In addition, in order to mitigate category over-use
and other forms of awareness bias, we can
exclude any panelist from a clients survey by topic

of survey recently taken, frequency of

participation, or for tracking studies participants
from prior waves.
Finally we have developed a sampling technique
termed SmartSelect, that can be used to further
ensure that respondents are representative of the
desired target population. In short, SmartSelect
compares the incoming sample to a baseline and
allows respondents into the survey as long as they
match what is in the baseline sample. If an
incoming piece of sample is determined to be over
represented based on what is called their
propensity score, the respondent is not allowed
into the survey. The sampling process continues
until the incoming sample matches the baseline
sample. If the baseline sample is a high quality,
representative sample, the incoming sample
borrows those properties and is high quality and
representative as well. If further information is
needed about this technique, please see the
following paper.
International Journal of Market Research: A smarter way to select respondents for surveys?
Q8. Do you employ a
Context: A survey router
survey router?
is a software system that
allocates willing respondents to surveys for which
they are likely to qualify. Respondents will have
been directed to the router for different reasons,
perhaps after not qualifying for another survey in
which they had been directly invited to participate,
or maybe as a result of a general invitation from the
router itself. There is no consensus at present about
whether and how the use of a router affects the
responses that individuals give to survey questions.

Quota Sampling - As
demographic breaks
are established, we
create quotas for the
necessary number
of respondents that
meet specific criteria. Surveys traffic is
distributed and routed to achieve these quotas.
Targeted Screening - Pre-screening
respondents to participate in surveys based
on demographic, attitudinal or behavioral
information.
Census Outgo At times, clients are looking
to replicate census or another set of
parameters on outgo and do not want quotas
set to manage incoming survey data, for
example, a census distribution of respondents
touch their survey and dont necessarily
complete it.

If a participant is screened out of a survey they will

have the chance to re-enter the survey router and
attempt to qualify to take another survey.
Using a survey router ensures that respondents
that wish to participate in surveys, can do so
(before survey routers were used, often
respondents would fail to qualify for a survey theyd
been invited to because quotas were full, the
survey closed, etc.) and that multiple respondent
sources can be used without introducing source
bias.

Toluna does employ a survey router, or sample

allocation engine however it can be deactivated on
a project level if requested by a client or if we
decide to do so of our accord for a variety of
possible reasons. The sample-allocation engine
can allocate respondents to projects using filters or
specific methods for sampling, including:

Q9. If you use a

router: Please describe
the allocation process
within your router. How
do you decide which
surveys might be
considered for a respondent? On what priority
basis are respondents
allocated to surveys?

Context: Biases of
varying severity may
arise from the
prioritization in
choices of surveys to
present to
respondents and the
method of allocation.

Tolunas survey router is

described in the
response above.
Toluna was one of the first companies to introduce
a survey router as a means of ensuring that the
right respondent, gets to the right survey, at the
right time. Routing is now considered to be standard practice.
Tolunas router is proprietary and assigns respondents randomly, with differential probabilities, to
surveys that appear to be good fits. Routing can
be used with every sample source (panel, RealTime Sampling, third party providers, etc.) and
in conjunction with all sample selection methods
(quota sampling, targeted sampling and census
outgo are mentioned above).
Again, one of the primary benefits of using a router
is to ensure that multiple respondent sources can
be used without introducing source bias.
Context: If Person A is
allocated to Survey X on
the basis of some characteristic then they may
not be allowed to also
do Survey Y. The sample
for Survey Y is potentially
biased by the absence of
people like Person A.

Q10. If you use a

router: What
measures do you take
to guard against, or
mitigate, any bias arising
from
employing a router?
How do you measure
and report any bias?

Respondents are randomly assigned to surveys

for which they might qualify. This practice helps
prevent the bias noted above as person A is
randomly assigned to a survey of the many open
surveys for which they might qualify.

Further, in cases where

clients seek an iron-clad
approach to ensuring
respondent
representativeness, we
will recommend our
proprietary SmartSelect,
sampling methodology.
Q11. If you use a router:
Who in your company
sets the parameters of
the router? Is it a
dedicated team or
individual project
managers?

Context: It may be
necessary to try to
replicate your project in
the future with as many
of the parameters as
possible set to the same
values. How difficult or
easy will this be?

Tolunas router is managed by the supply
organization; however individual fieldwork project
managers do have the ability to limit the
percentage of sample generated by router traffic
on a project-by-project basis. Any changes to the
router behavior is approved and tested by Tolunas
Chief Research Officer to ensure consistency of
data.
Context: The
usefulness to your
project of pre-profiled
information will depend
on the precise question
asked and may also
depend on when it was
asked. If real time
profiling is used, what
control do you have over
what question is actually
asked?

Q12. What profile data

is held on respondents?
How is it done? How
does this differ across
sample sources? How is
it kept up-to-date? If no
relevant profiling data is
held, how are low
incidence projects dealt
with?

Toluna collects up to 1,700 current attributes

through a number of specialist screeners. The
1,700 are the key profile attributes, for which we
collect as many as 750 completed per panelist.
Profiles of Tolunas specialist panels are optimized

for a specific market sector as follows:

Automotive
Beauty
Finance
Home and Family
Internet
Media
Shopping
Sports
Travel
Video Games
Mobile
B2B/Employment
Medical Professionals
Health

Registration
Tolunas profiling process is continuous the
objective being to have broad and current
attributes on each member. Each member has a
personal dashboard and administration area within
their Toluna.com account, and a dashboard
indicator to help them manage their personal
profile.
Members earn points by completing and
updating their profile. This system ensures that
Tolunas panelists are rewarded, motivated and
enabled to continually monitor the depth and
accuracy of their profiling information. As a result,
Toluna can target its survey invitations extremely
accurately and our clients enjoy consistently
impressive response and completion rates.
Regarding specialist profilers, across all countries
and members, the average Toluna panelist
completes above 50% of additional profile
information for that county.

Q13. Please describe the

survey invitation process.
What is the proposition
people are offered to
take part in individual
surveys? What
information about the
project itself is given
in the process? Apart
from direct invitations
to specific surveys (or
to a router), what other
means of
invitation to surveys are
respondents exposed to?
You should note that not
all invitations to
participate take the form
of emails.

Context: The type of

proposition (and
associated rewards)
could influence the type
of people who agree to
take part in specific
projects and can
therefore influence
sample quality. The level
of detail given about
the project may also
influence response.

Tolunas proprietary panel management platform

enables fieldwork managers to accurately estimate
the number of available panel members with
characteristics that match a clients desired target
profile. Based on internal guidelines relating to
historical response (for example by age, gender
and country of residence), a random sample is
then selected.
That fieldwork manager is responsible for the
mailing of invitations and managing filling of
required quotas. The invitation contains incentive
information (dependent on length of survey and
target group), and any project timelines along with
approximate length of survey together with a link
to the survey and the entire URL, which recipients
can paste into their browser so as to ensure
there are no issues with pop-up blockers. No
information linked to the target sought is included
in the invitation so as not to introduce a bias in
respondent answers to screening questions.

The following is a list of the ways panelists can be

invited to participate in surveys:

Direct Invitations, or Targeted Auto

Invitations - Respondents are selected for specific surveys based on the match of their profile
information with the survey target, for precise
targeting and on-time study
completion. Invitations are delivered via email,
mobile text, or through the Toluna App, with a
link directly into a specific survey. Direct
Invitations are sent by a project manager,
whereas Targeted Auto Invitations are deployed
via Tolunas sampling system.
My SurveyCenter Invitations - At My Survey
Center, an area within Toluna.com, Toluna
panelists can participate in surveys on-demand.
Available surveys are listed on each panelists
personal web portal. Similar functionality is
available to users of the Toluna App.
Real-Time Sampling - Respondents view
invitations within websites, and these can be
banner advertisements, text links, and more.

All respondents can be routed if they do not qualify

for the survey theyve been invited to participate in.

Q14. Please describe the

incentives that respondents are
offered for taking part in
your surveys. How does
this differ by sample
source, by interview
length, etc?

Context: The reward or

incentive system may
impact on the reasons
why people participate
in a
specific project and
these effects can cause
bias to the sample.

Tolunas standard incentive process is a pointsbased system. Incentive levels are determined
according to the following factors: subject matter,
commitment (i.e., length of interview required), and
incidence. Note that if the respondents are joining
a panel and/or will participate in repeat surveys,
an incentive would be appropriate to maximize
response rates at each stage. If a client requires a
custom incentive process, Toluna is capable of

implementing one in
order to ensure that the
incentive does not bias
the clients
measurements. Toluna
typical incentive system
is one in which panelists earn points for each
survey completed. These points are automatically
redeemed as vouchers.
Tolunas panel management system is expert at
accounting for incentive accumulation and
redemption for points-based reward programs,
sweepstakes awards, and prize distribution.
Points
Points are earned for completing a survey. Points
are also rewarded for answering a sponsored
polll. These polls differ from user-generated polls,
as they are created by Toluna on behalf of clients
wishing to gain instant top-line research data or to
assess project feasibility on a particularly niche
target profile. Points are also accumulated by
members for completing their profiling
questionnaires, and for regularly updating them.
Rewards
Our points-based incentive system enables
members to use their points to exchange for
vouchers and gifts from our reward partner
network which is clearly highlighted to all
members. Our program also includes Gifties,
which enables panelists to redeem points for gifts
that have a chance of becoming real.
Prize draws
Members can also enter into multiple prize draws
for cash and products. Points earned can be
exchanged for prize draw tickets.

10

Context: The size of any

panel or source may not
necessarily be an accurate indicator that your
specific project can be
completed or completed
within your desired time
frame.

Q15. What
information about a
project do you need in
order to give an accurate
estimate of feasibility using your own resources?

To run feasibility on a potential project Toluna

requires job specifications for the total sample size
desired, target audience, questionnaire length,
and projected incidence vis a vis the population
of interest, and expected response rate. With this
information our fieldwork management staff can
predict with a high degree of accuracy the internal
feasibility of a project, and realistic timelines for
completion.
Q16. Do you measure
respondent
satisfaction? Is this information made available
to clients?

Context: Respondent
satisfaction may be an
indicator of
willingness to take future
surveys.
Respondent reactions to
your survey from self-reported feedback or from an
analysis of suspend points might be very valuable to
help understand survey results.
Toluna has a quarterly panelist Satisfaction Survey
across all panels (yearly in some markets)
reflecting our belief that each member is an
important asset. Panelist satisfaction
questionnaires ask members to provide feedback
on:

Surveys
Email invites
Newsletters
Incentives
Web usability
Support quality

Q17. What
information do you
provide to debrief your
client after the project
has finished?
Context: One should
expect a full sample provider debrief report, including gross
sample, start rate, participation rate, drop-out rate,
the invitation/contact text, a description of the field
work process, and so on. Sample providers should
be able to list the standard reports and metrics that
they make available.
For every project we provide the following as
standard:
TolunaAnalytics provides live fieldwork information
for Toluna projects (excluding Sample Only) and
provides real-time feedback to clients.
We close projects when we have confirmation that
we have met all specified criteria, including
number of completes (overall and per quota cell).
Upon request we can provide: completion rates,
respondent dropout rate, response rate per cell/
target profile, total number of invitations sent,
number of screen outs, timings of email delivery
and responses, number of reminders sent during
fieldwork.
Each project concludes with a Client Satisfaction
survey, and a call from the client service manager
to appraise the success of the project in terms of
data, timeline, process and client experience.

11

Data Quality & Validation

Q18. Who is
responsible for data
quality checks? If it is
you, do you have in place
procedures to reduce or
eliminate undesired
within survey
behaviors, such as (a)
random responding, (b)
Illogical or
inconsistent
responding, (c) overuse
of item non-response
(e.g. Dont Know), or (d)
speeding (too rapid survey completion). Please
describe these procedures.

Context: The use of such

procedures may increase
the
reliability and validity of
the survey data.
Tolunas dedicated
quality management
team use a variety of
different techniques for
monitoring and limiting
fraudulent respondents
through active cleaning
and exclusion of
observed offenders from
the panel.
Inactive, unresponsive,
inattentive and
fraudulent panelists

are regularly eliminated from the panel.

Fraudulent panelists are put in a blacklist (to avoid
any possible re-registration). Under no
circumstance will these members be sent an
invitation email to complete a survey or are
included in the size of our panels.
During a project our team will run systematic
checks to ensure the quality of data generated for
our client is of the highest standard. We include
open end response checks for most languages
and consistency checks on demand for speeding,
straight-lining and red herring. The following are
performed as standard:
Speed-checks are applied to any user who
completes a survey in less than 1/2 of the
median survey duration.
Straight-lining can be applied to any grid
question with at least 4 scale levels and 15+
items (Toluna validates with clients the
appropriateness of these checks on a
question-by-question basis).

Red herrings
(trap questions) are
used in most
questionnaires to
validate the respondent attention.
In addition, we can accommodate other trap
questions that compare prior responses within
survey, or demographic questions against profile
data.
Q19. How often can the
same individual be contacted to take part in a
survey within a specified
period whether they respond to the contact or
not? How does this vary
across your sources?

Context: Over
solicitation may have an
impact on
respondent
engagement or on
self-selection and nonresponse bias.

Frequency of contact:
We aim to contact panelists regularly, without
burdening them. Our system allows respondents to
be contacted with survey invitations daily, although
we do so, in practice, less frequently. This excludes
reminders, and other member communications.
Participation in surveys:
On average over the course of a respondents
lifetime, most respondents take an average of 8.7
surveys.
Email reminders:
Email reminders => the maximum number of
reminders is set to 2 reminders per project and 24
hours between Invite and reminder 1 and 24 hours
between reminder 1 and reminder 2.
Context: Frequency of
survey
participation may
increase the risk of undesirable
conditioning effects or
other potential biases.

Q20. How often can the

same individual take
part in a survey within a
specified period? How
do you manage this
within categories and/or
time periods?

The average panelist completes 8.7 surveys over

the course of their lifetime with Toluna.
12

Q21. Do you maintain individual level data such

as recent
participation
history, date of entry,
source, etc., on your
survey respondents? Are
you able to
supply your client with a
project
analysis of such
individual level data?

Context: This type of

data per respondent
including how the total
population is defined
and how the sample was
selected and drawn, may
increase the possibilities for analysis of data
quality.

Our proprietary panel

and community
management system
enables us to record a complete and automatically updated history of all panelist participation
and interaction. We record such data as recruitment source, incentives awarded/claimed, emails
opened, surveys taken, profiling data and questions answered. This data can be provided to
clients on a batch level to retain respondent
anonymity.
Context:
Q22. Do you have a conConfirmation of
firmation of
identity can increase
respondent identity
quality by
procedure? Do you have
decreasing multiple enprocedures to detect
tries, fraudulent panelists fraudulent respondents?
etc.
Please describe these
procedures as they are
See Q3 for information
implemented at sample
about Tolunas
source registration and/
SmartSample data
or at the point of entry to
quality program and
a survey or router. If you
means of ensuring
offer B2B samples what
respondents are Real,
are the procedures there,
Valid, Engaged,
if any?
De-duplicated, and
Representative.
Information from that
prior response is also
included below.
All Toluna respondents are authenticated.
Respondents are subject to GeoIP and validated
against Postal Address File (PAF) (Melissa data in
the US), a CAPTCHA confirmation process, and a

third-party technology
Imperiums Address
Correction, which
prevents individuals
from participating in
surveys unless a valid
postal address is used.
Panelists must complete Tolunas double-opt in
registration processes, which requires individuals
to confirm membership by clicking on an emailed
link upon enrollment in the Toluna panel.
All respondents are validated using email address
validation measures (Toluna blocks known fake or
disposable IP domains).
TrueSample validation is available for respondents
in the U.S., Canada, UK, Germany, France, Spain
& Italy. For our proprietary healthcare
professional panels in the UK, France, Germany,
Italy and Spain, we collect web urls and phone
numbers for the physicians practice to validate.
Our US physicians are American Medical Association (AMA) member validated when they first join,
and again each time they attempt to take a survey.
De-duplication
Respondents cant enroll in the Toluna
community, or take a survey more than once,
fraudulently, or accidentally. To prevent this from
happening, Toluna uses a Match 2 process to flag
similarities amongst Toluna panelists upon
enrollment. This automated process clusters
individuals and helps to identify members that may
have attempted to register as members of the
Toluna panel more than once. In addition to this
process, Toluna has developed Duplicate
Respondent Detection technology. This cookiebased technology is used during the panelist
registration process, and at the beginning of every
Toluna survey. This is coupled with a third-party
digital fingerprinting technology, Imperiums
RelevantID, which is again, used at the
beginning of every Toluna survey.
Tolunas process for preventing respondent

13

duplication is proactive, meaning that respondents

are prevented from participating in joining our
panel, or participating in surveys more than once
no matter how an individual is invited to participate
(via email, Real-Time Sampling etc).

Policies & Compliance

Q23. Please describe
the opt-in for market
research processes for
all your online sample
sources.

Context: The opt-in process indicates the

respondents
relationship with the
sample source
provider. The market
generally makes a distinction between single and
double opt-in. Double opt-in refers to the process
by which a check is made to confirm that the person
joining a panel or database wishes to be a member
and understands what to expect (in advance of participating in an actual survey for a paying client).
New panelists are required to double opt-in. The
process is as follows:
Step 1 - A prospective panelist completes a panel
registration form, which includes contact and
demographic information (first opt-in).
Step 2 - An automatic email is sent to the
prospect, requesting verification of their panel
registration by clicking a link that confirms their log
-in details.

Q24. Please provide

a link to your Privacy
Policy. How is your
Privacy Policy
provided to your
respondents?
Context: Not complying with local and
international privacy laws might mean the sample
provider is operating illegally. An
example privacy policy is given in the
ESOMAR Guideline for Online Research.
Toluna adheres to the globes most stringent
guidelines for panel management and data quality,
and respondent privacy.
The Toluna Panel Privacy Policy has its own page
on the panel website and is easily accessible from
any page on the site. A link to the Privacy Policy
is contained in all survey invitations as a standard
practice. The Policy complies with established
industry guidelines and states that members
personally identifiable information will never be
rented, sold, or revealed to third parties except in
cases where Toluna must share it in order to
comply with the law.
The full policy can be viewed on our member
website www.toluna.com.

Step 3 - Once the prospect has clicked the link

(second opt-in), he or she is officially a panelist and
is presented with an opportunity to complete
additional profiling.

14

Q25. Please describe the

measures you take to
ensure data protection
and data security.

Context: The sample

provider usually stores
sensitive and
confidential information
on panelists and clients
in databases. These data
need to be properly secured and backed-up, as does
any confidential information provided by the
client. The sample provider should be able to
provide you with the latest date at which their
security has been evaluated by a credible third-party.
With respect to respondent confidentiality, the
council explicitly states the following: Since
individuals who are interviewed are the lifeblood of
the Survey Research Industry, it is essential that
Survey Research organizations be responsible for
protecting from disclosure to third parties
Including clients and members of the publicthe
identities of individual respondents, as well as
respondent-identifiable information, unless the
respondent expressly requests or permits such
disclosure. Accordingly, Toluna executes best
practices by ensuring that respondents are not
associated with the personal identifying information
that they provide.

To ensure that data is not compromised in any
way, all personally identifiable information (PII) is
stored using one-way encryption techniques, and
passwords are stored using password keys.
Our data center is redundant ensuring data is
quickly recovered in the case of natural disaster.
Toluna also adheres to the Safe Harbor Agreement
as set forth by the U.S. Department of Commerce
regarding the collection, use, transfer and retention of data from the European Union to the United
States.

Q26. What practices do

you follow to
decide whether
online research should
be used to present commercially sensitive client
data or materials to
survey respondents?
Context: There are no foolproof methods for protecting audio, video, still images or
concept descriptions in online surveys. In todays
social media world, clients should be aware that the
combination of technology
solutions and respondent confidentiality agreements are speed bumps that mitigate but cannot
guarantee that a clients stimuli will not be shared or
described in social
media.
Toluna panelists are informed that they may be
presented with confidential material in the terms &
conditions when they first join the panel, and they
are asked to agree not to share concepts that they
are shown in Toluna surveys. We can also make
this an opt-in process on an individual survey, if
necessary.
In cases where an ironclad approach to security
and confidentiality is required we leverage
SafeView is a Java-based program that ensures
additional concept security vs. the standard
data quality checks. Mouse right-click, saveas, printing, Windows Clipboard, cache storage,
screen capture software (i.e. SnagIt) are all
disabled with SafeView. This technology prohibits
survey respondents from saving images, video or
audio files locally, or forwarding them to others.

We have an uptime rate of 99.97 percent.

Scheduled maintenance and feature upgrades are
announced 48 hours in advance, and completed
during off-hours.

15

Context: Being
Q27. Are you
certified may
certified to any
require the supplier to
specific quality
perform tasks in a presystem? If so, which
determined
one?
manner and document procedures that should be
followed.
Toluna has a dedicated in-house team dedicated
to quality management. This group defines, trains
staff and measures against our quality processes.

Our quality management procedures are detailed
below for both providing sample and for
programming clients questionnaires. Our company
operations are governed by documented
processes, including those for project
management, survey programming, quality control,
sampling, panel management, and confidentiality
and security of information provided by clients,
panelists, and other proprietary records. These
processes are reviewed and updated on a
continuous basis. We have reviewed the proposed
ISO standards for market research access panels
and have found our processes to be substantially
in compliance.
Quality Checking
At Toluna we believe quality starts at the early
stages of project management, by ensuring all
parties have a clear understanding of the research
objectives and client requirements.

Toluna is committed to delivering data of the
highest quality, ensuring that the final results of the
research are meaningful, client specifications on
programming, data collection, and data processing
are met, and the overall client experience with the
Toluna Project Management and Operations teams
is smooth.

The following is a brief

overview of the quality
checks many
automated that are
performed on the
execution of every
operations project at
Toluna:
Internal review of the overall project
specifications
including a review of project details discussed
between client & Toluna sales team
Toluna review of the final questionnaire
including a review of questionnaire logic,
qualifying criteria, and termination conditions
Toluna and client review questionnaire and
overall project details, including clarifying logic
of the questionnaire plus a review of
programming and data requirements.
Survey programming then performs the
following quality checks:
Survey Programming personnel do two to
three runs of peer-to-peer manual and
random data generation (RDG) testing of
the survey links programmed, and any
programming issues identified at this stage
are sent back to the programming team and
are fixed by the programmers;
The project manager does final manual
test before sending it to the client and any
programming issues identified at this stage
are sent back to the programming team and
fixed by the programmers;
Client reviews the programmed survey, and
notes if any changes are required;
Project manager reviews changes and
sends the survey to Survey Programming
for the final changes; and
Final testing is then confirmed by the
Project Manager and the client.
Before field launch, and after receiving final
sign-off from client, Toluna does a final test
using automated testing, through which
randomly generated data is tested to verify the
logic.

16

 The fieldwork manager also double-checks

quotas and tests the survey invitation before
sending sample.
After field launch, the Data Processing team
processes partial data files and tests the data
for errors, formats, file types, and so on.
Client Satisfaction Survey
Toluna has an independent Quality Team charged
with continual, real-time mapping, measuring,
and improving processes through Sig Sigma Lean
methodology. The Quality Team approves and
monitors process changes, eliminating the
possibility of adverse effects. On a 24/7 basis, the
Quality Team is monitoring and reporting on the
current health of the business, while converting
those reports into recommendations and actions
that will deliver continuous improvement.
The team has developed two main tools to fulfill its
objectives. The first is a client-satisfaction survey.
For each project we service at Toluna, we send
out a short questionnaire to clients, in order to assess our performance. The second tool is a set
of defined KPIs (Key Performance Indicators) that
are backed by a real-time reporting system. The
system reports on three main areas, defined below.
Business Metrics, those indicators are that
are client-impacting. These are high-level and
reflect the business health of Toluna and focus
on defective percentages, areas where we
perform contrary to client expectations. The
goal is to improve the what we say to what we
do ratio, in other words, improving delivery on
promises.
Functional Metrics are focused on internal
impact, those areas where the delivery is
internal and has ramifications for other
Operations groups. These metrics indicate
department health and mainly focus on quality
of internal deliverables and cycle time.
Process Metrics are focused on compliance
with standards and process quality.

Q28. Do you conduct

online surveys with children and young people?
If so, do you adhere to
the standards that
ESOMAR provides? What
other standards, for
example COPPA in the
United States, do you
comply with?

Context: The ICC/

ESOMAR International
Code requires special
permissions for
interviewing children.
These are described in the ESOMAR Online Research
Guideline. In the USA researchers must adhere to the
requirements of the Childrens Online Privacy Act
(COPPA). Further information on legislation and
codes of practice can be found in Section 6 of
ESOMARs Guideline for Online Research.
Toluna complies with all legal requirements and
industry guidelines, with respect to interviewing
children and young people outlined by COPPA, the
UE Safer Internet programme and ESOMAR.

Toluna will not knowingly collect any information
from any child without permission from his/her
parents or legal guardian. The minimum age
requiring adult permission varies by country.
Eligible young people are invited to participate
in our Teens panel. Toluna can advise any
client wishing to conduct research relating to
young people across Europe, Asia and North
America.

17

Appendix

2015

Esomar 28 { Appendix

Sample Quality Standards

Quality Checks to Ensure Quality Respondents
At Toluna we recruit members to the Toluna panel through various methods
including web-banners, website referrals, pay-per-click, natural search optimization,
affiliate marketing, email and online public relations activities using our SmartSample
process. SmartSample ensures that respondents are a) real, b) valid, c) unique, d)
engaged and e) representative.
A) Real: Respondents are GeoIP and postal code validated using Postal Address File (PAF) (Melissa data in the
US). Imperiums Address Correction product is used to prevent individuals from participating in surveys unless
a valid postal address is used. In addition, each respondent completes a CAPTCHA confirmation process. New
panelists are required to double opt-in. The process is as follows:
Step 1: A prospective panelists completes a panel registration form, which includes contact and
demographic information. This is the first opt-in.
Step 2: An automatic email is sent requesting verification of registration by clicking a link that confirms their
log-in details.
Step 3: Once the prospect has clicked the link he or she is officially a panelists. This is the second opt-in.
B) Valid: Toluna blocks individuals with known fake or disposable domains. When possible, respondents are further validated using the TrueSample validation process, which compares an individuals name and address with
third-party sources. TrueSample validation is available for respondents in the U.S., Canada, UK, Germany, France,
Spain & Italy. For our proprietary healthcare professional panels in the UK, France, Germany, Italy and Spain, we
collect web URLs and phone numbers for the physicians practice to validate. Our US physicians are AMA (American Medical Association) member validated when they join, and each time they take a survey.
C) Unique: Respondents cannot enroll in the Toluna community, or participate in the same survey more than once,
fraudulently or accidentally. Tolunas Duplicate Respondent Detection cookie-based duplication-detection
technology is used during the panelist-registration process, and at the beginning of every Toluna survey. This is
coupled with a third-party digital fingerprinting technology, Imperiums RelevantID, to prevent respondents from
participating in surveys more than once. As a further measure to prevent duplication, Toluna uses technology to
produce a proprietary, matching algorithm to flag similarities of new registrants with existing Toluna panelists.
D) Engaged: Following are some of the checks utilized to ensure respondent engagement:
timestamp terminates to remove speeders
straightline checks, red herrings and coherence checks to ensure attentiveness
To see a detailed list of our data quality checks ask for Tolunas Data Quality Standards document
E) Representative: Representativeness is achieved by selecting a sample that reflects the population-of-interest accurately. This can be achieved through the appropriate application of simple or stratified random sampling,
quota sampling, or by using more advanced techniques such as SmartSelect. SmartSelect is a selection
method that ensures the characteristics of each respondent for each project, irrespective of sample source, match
those of the population of interest.
In addition to using appropriate sampling techniques, representativeness can be achieved by weighting the data
to be representative of the population-of interest. Toluna experts understand the appropriate weight function to
apply and appropriate variables to use in the weighting algorithm.

19

Data Quality Standards

Quality Check

How it works

Application

Duplicate
Respondent
Prevention

To ensure respondents do not take your survey more than once,

either fraudulently or accidentally, we use a proprietary technology
called Duplicate Respondent Detection (DRD) with Toluna.com
panelists and respondents from our affiliate network of partners.
DRD is coupled with a third-party digital fingerprinting technology to
ensure respondents can only complete your survey once.
After the soft launch has been completed the median interview
length is determined. Respondents who completed the survey in
less than 50% of the median interview duration are classified as
speeders and removed from the data set. An automated terminate
status will be implemented for the remainder of fielding to ensure
your data is comprises thoughtful responses by engaged
respondents.
To ensure a respondent who has taken a recent version of the
same survey we can exclude respondents based upon their status
in another survey.

All surveys fielded by

Toluna

Speeders

Participation Status
Exclusion

Straightline
Prevention

Red Herring
Questions

Coherence Checks

Quality checks
applied to each
survey
Cleaning Scripts

Open End Checks

Some respondents may apply the same ratings for a series

of attributes in grid questions, thereby straightlining to complete
the question with minimal effort. Checks can be applied to relevant
grid questions with at least 15 attributes and four or more scale
levels. Respondents are flagged if the standard deviation is less
than 0.5.
These are questions added to the survey to ensure the
respondent is paying attention to the questions being asked. Red
herring questions involve asking the respondent to select a specific
answer option to confirm that they are paying attention.

All surveys

Trackers, multiple
surveys with similar
or recent fielding
dates
Full Service surveys
with relevant grid
questions

Full Service surveys

with questions
containing long
answer lists, or long
duration
To ensure respondents are providing thoughtful answers, questions Full Service surveys
can be repeated at different points in the survey. If the responses
with respondents
are different for each version of the question we know the
who have a special
respondent was not attentive, and they can be removed from the
or specific
data set.
experiences, etc
Manual checks of the survey link are completed by members of the All Full Service
programming team, and then by the Project Manager before being Surveys
sent to you for review. Random data generation (RDG) testing is
conducted prior to the survey being launched in field.
Data cleaning is the process by which we correct errors and
All Full Service
anomalies in the case data. To do this we write cleaning scripts to
Surveys
ensure respondents followed the correct path through your survey.
Respondents who consistently respond to open ended
All Full Service
questions with nonsensical or offensive entries may be doing so
Surveys with open
due to lack of engagement. Our data processing team will flag such end questions
respondents and remove them if deemed appropriate to do so.

20