Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning

For overparameterized optimization tasks such as those in modern machine learning, global minima are generally not unique, so understanding generalization requires knowing which minimum an optimization algorithm converges to. Minima that are unstable under the dynamics of the algorithm restrict the minima it can find. In joint work with Maximilian Engel (University of Amsterdam), Dennis Chemnitz characterizes the global minima that are dynamically stable or unstable for both deterministic gradient descent and stochastic gradient descent (SGD). The talk introduces a characteristic Lyapunov exponent that depends on the local dynamics around a global minimum and presents a rigorous proof that the sign of this exponent determines whether SGD can accumulate at that minimum.