To survive, organisms must make rapid inferences about their surroundings. For example, to successfully escape approaching predators, prey must estimate the direction of approach from noisy stimuli [1]. Such rapid inferences are particularly challenging because there is little time to gather sensory information, yet the precision of inference is critical for survival. Due to evolutionary pressures, nervous systems likely evolved effective computational strategies that enable accurate inferences under strong time limitations. Here, we aim to understand the principles underlying such rapid inferences. We consider a specific instantiation of a general scenario in which an observer must determine the direction of an approaching predator from a brief sequence of noisy sensory signals. Traditionally, the relationship between the duration and accuracy of inference is described by a "speed-accuracy tradeoff" (SAT). Intuitively, the longer the observer collects sensory information, the more accurate the resulting inference. Here, we deliberately consider scenarios where the observer is faced with strong time constraints, and thus cannot reliably perform accurate inference. Our key insight is that while the SAT characterizes performance on average, it can be decomposed into diverse patterns of error dynamics on individual trials. Each pattern is generated by a different stimulus sequence that occurs by chance. These can be separated into "optimistic" sequences that permit rapid and accurate inferences, and "pessimistic" sequences that generate high errors that persist over time. To exploit this single-trial structure, we extend the standard Bayesian inference setting by deriving adaptive stopping rules that rely on the evolving dynamics of posterior uncertainty. The resulting inferences violate the SAT and confer several advantages, including increasing the average speed and accuracy of inference and thereby improving the probability of successful escape. Finally, we show that the adaptive observer qualitatively reproduces features of escape behavior in Drosophila melanogaster [2] whose escapes are highly optimized.