Privacy-Preserving Methods for Repeated Measures Designs

Evolving privacy practices have led to increasing restrictions around the collection and storage of user level data. In turn, this has resulted in analytical challenges, such as properly estimating experimental statistics, especially in the case of long-running tests with repeated measurements. We propose a method for analyzing A/B tests which avoids aggregating and storing data at the unit-level. The approach utilizes a unit-level hashing mechanism which generates and stores the first and second moments of random subsets of the original population, thus allowing estimation of statistics, such as the variance of the average treatment effect (ATE), by bootstrap. Across a sample of past A/B tests at Netflix, we provide empirical results that demonstrate the effectiveness of the approach, and show how techniques to improve the sensitivity of experiments, such as regression adjustment, are still feasible under this new design.