Encouragement Designs: An Approach to Self-Selected Samples in an Experimental Design

Abstract

When subjects choose their own treatment (are self-selected into treatments based on their varying compliance with assignment to treatment states), many of the well-developed techniques of randomization-based experimental design and analysis are no longer applicable. With random assignment to treatment and control conditions, we can reasonable assume that over many replications, the two groups will be similar in all respects (observed and not observed) but that of receiving the treatment. With non-random assignment, it may be true that the groups differ on the variable of interest prior to receiving the treatment, and that a measured post-treatment difference (or lack thereof) will be erroneously attributed to the treatment. Our approach involves making all subjects aware of the availability of the treatment, but then offering extra encouragement (an encouragement design) to participate in the treatment to a randomly selected half of the population. If the encouragement is successful, we show how this leads to an estimable treatment effect (with associated asymptotic standard error).

An illustrative example using fictitious data is presented in which we measure the effects of coupon books on store purchase behavior. A randomly selected half of the customers at a given store are assigned to receive coupon books as part of a package of store materials (the treatment condition) whereas the other half are not. We overlay on this initial randomized design an encouragement condition in which a second randomly chosen half of the store's customers receive their materials from an employee (encouraging participation with the treatment), and for the other half the materials were simply placed on a table. The market share of persons who chose to take the coupon book was 44% and 20% from those who did not. A naive estimate of the effect of the treatment is 24%; however this estimate ignores the self-selection bias (those persons assigned to receive the coupon book but refused to take it, and those not assigned who picked one up anyway). Using the encouragement design estimator derived leads to a lower estimated impact of the treatment at 5%. The difference in these results may have significant impact on the choice by store management to utilize employees to hand out promotional materials.