Discrete communication mediates effective regularization in recurrent neural networks

Neuronal computation is mediated by spikes, yet it is unclear what the benefits of discrete spiking dynamics over continuous firing rates are. Many theoretical and computational studies treat single neurons as continuous units and their output as an effective firing rate. These works view spiking dynamics as a biophysical constraint, e.g., for energy efficiency. Conversely, other theories suggest that the exact timings of single spikes are meaningful. We propose a novel theory that shows the benefits of spiking dynamics on neural computation. In particular, we show that spiking neural dynamics can improve generalization, even when the information is encoded in the averaged firing rates and not in individual spikes.
We derive a mean-field theory for large networks of continuous and discrete (binary) neurons in the thermodynamic limit. The readout from the networks is expressed as a Gaussian process shaped by a time-dependent kernel. This function describes how the similarity between a pair of inputs is trans- formed into the similarity of the corresponding network states at later times. We find that the neural codes described by the respective kernels of the discrete and continuous networks have qualitatively different features, even when all other parameters are identical. In particular, discrete networks show strong microscopic chaos at short time scales while maintaining long-range correlations. The local chaos results from the discontinuity in the transfer function of the discrete neurons and is different from that of continuous firing-rate models. We show that this fast divergence allows for regularization, improving readout generalization. Overall, we demonstrate that spiking dynamics can mitigate the overfitting of recurrent networks to noisy data. Our theory contributes a novel explanation of spiking dynamics’ possible benefits in cortical circuits.