Cortical neurons exhibit rich, time-dependent patterns of activity that are often selective to specific stimuli. From a dynamical systems perspective, such complex activity can be conceptualized as transient amplification. It is still unclear if transient amplification is an artifact of a carefully tuned model, or a naturally occurring phenomenon. Moreover, minute changes in the connectivity of amplifying networks can cause catastrophic dynamic instabilities. How biological circuits could settle into stable, yet amplifying connectivity regimes in an ever-changing brain is unknown. Here, we show that transient amplification is a ubiquitous network quality that can be achieved without fine-tuning. We use Schur decomposition in linear rate models to find the space of all Dalean and stable connectivity matrices (DS). We show that the fraction of amplifying solutions increases both with connectivity strength and network size, revealing a predominantly amplifying DS. Next, we study the trade-off between richness of dynamics and robustness to connectivity perturbations. For this, we consider a simpler system with one excitatory and one inhibitory unit. We can derive analytically the space of Dalean, stable, and amplifying matrices (DSA). We show that the most amplifying networks are also closest to the instability boundary. Interestingly, circuits can safely traverse DSA (i.e., learn) without risking catastrophe by way of homeostatic constraints that prevent crossing the instability boundary. Our findings argue for the biological plausibility and ubiquity of transient amplification and show how amplification constrains the nature of allowable connectivity changes in neural systems.