Synaptic Neural Balance

Pierre Baldi develops a general theory in which a neuron is balanced when its input and output weights have equal regularization cost. The theory extends beyond ReLU networks with L2 penalties to other activations, Lp regularizers, recurrent and convolutional architectures, and mixed networks. Ordinary gradient descent need not preserve balance; regularized learning should converge toward it. Local scaling and balancing operations lead to a stochastic algorithm whose unique global solution follows from strictly convex optimization on an architecture-dependent linear manifold. Simulations suggest that balancing before or during training improves speed and performance. Connections to overparameterization, group symmetries and neurobiological or neuromorphic networks are discussed.