Publications
Publications
- 2024
- HBS Working Paper Series
Winner Take All: Exploiting Asymmetry in Factorial Designs
By: Matthew DosSantos DiSorbo, Iavor I. Bojinov and Fiammetta Menchetti
Abstract
Researchers and practitioners have embraced factorial experiments to simultaneously test multiple treatments, each with different levels. With the rise of technologies like Generative AI, factorial experimentation has become even more accessible: it is easier than ever to generate different versions of potential treatments. Typically, in large-scale factorial experiments, the primary objective is to identify the treatment with the largest causal effect. This is especially true for experiments that suffer from measurement error, attrition, non-compliance, and censoring: point estimates are unreliable, but—as we show—the asymmetry in the largest treatment effect makes it possible to identify the most impactful treatment even when point estimates are biased. To exploit this asymmetry, we propose using a Fisher randomization test as a general non-parametric approach for inference, which we apply to an existing field experiment that measured intern performance at a large financial firm. We show that the earliest possible intervention has an immediate and enduring impact: performance improves in the week of the intervention and in future weeks, sometimes even to a greater extent than interventions in those future weeks. The takeaway—intervene early—has important consequences across the many contexts of workplace programs.
Keywords
Factorial Designs; Fisher Randomizations; Rank Estimators; Employer Interventions; Causal Inference; Mathematical Methods; Performance Improvement
Citation
DosSantos DiSorbo, Matthew, Iavor I. Bojinov, and Fiammetta Menchetti. "Winner Take All: Exploiting Asymmetry in Factorial Designs." Harvard Business School Working Paper, No. 24-075, June 2024.