Seminar, Hosted by Online Causal Inference Seminar
AI-assisted design and analysis of experiments with unstructured treatments
Eli Ben-Michael and Naoki Egami
Carnegie Mellon University; MIT
Abstract
Randomized experiments increasingly use complex, unstructured treatments such as text, images or video, often to estimate the effect of a structured, interpretable attribute, such as a linguistic or visual feature, that should generalize beyond the treatments used. Such studies face two problems: effects of the attribute may be confounded by other correlated attributes, weakening internal validity, and may not generalize beyond the experimental treatments, weakening external validity. Eli Ben-Michael presents a framework that uses generative AI or large databases to sample treatments so as to navigate this trade-off explicitly, shows how to conduct conservative asymptotic inference with a difference-in-means estimator, and develops a calibrated AI-assisted estimator that incorporates zero-shot AI predictions to improve precision without introducing bias. The analysis works in a finite-population framework in which treatment assignment is the only source of randomness and AI generations and predictions are treated as black boxes without performance guarantees, and is illustrated with two hypothetical experiments, one with text and one with images. Naoki Egami is the discussant.
Topics
- Randomized experiments, broader topic: experiments
- unstructured treatments, search World Wide
- generative AI
- large databases, search World Wide
- internal validity, broader topic: validity
- external validity, broader topic: validity
- difference-in-means estimator, search World Wide
- asymptotic inference, broader topic: inference
- zero-shot AI predictions, broader topic: AI predictions
- finite-population framework, search World Wide