Skip to contentSkip to filters

We use cookies for analytics.

AI-assisted design and analysis of experiments with unstructured treatments

Eli Ben-Michael and Naoki Egami

Carnegie Mellon University; MIT

Abstract

Randomized experiments increasingly use complex, unstructured treatments such as text, images or video, often to estimate the effect of a structured, interpretable attribute, such as a linguistic or visual feature, that should generalize beyond the treatments used. Such studies face two problems: effects of the attribute may be confounded by other correlated attributes, weakening internal validity, and may not generalize beyond the experimental treatments, weakening external validity. Eli Ben-Michael presents a framework that uses generative AI or large databases to sample treatments so as to navigate this trade-off explicitly, shows how to conduct conservative asymptotic inference with a difference-in-means estimator, and develops a calibrated AI-assisted estimator that incorporates zero-shot AI predictions to improve precision without introducing bias. The analysis works in a finite-population framework in which treatment assignment is the only source of randomness and AI generations and predictions are treated as black boxes without performance guarantees, and is illustrated with two hypothetical experiments, one with text and one with images. Naoki Egami is the discussant.

Topics