Learning in neural networks with brain-inspired geometry

There are many differences between artificial and biological neural networks. However, some of those less frequently considered are the differences between their respective weight distributions and dynamics. Here, we take inspiration from neuroscience and train networks with a multiplicative update algorithm called exponentiated gradient descent (EG). In comparison to gradient descent (GD), this corresponds to changing geometries and measuring parameter differences with the un-normalised relative entropy function instead of Euclidean distance.
In line with biology, the multiplicative updates prescribed by EG are unable to change weight polarity, and as such we show a feed-forward inhibition-inspired “Dale’s ANN” (DANN) architecture learns better with EG than standard networks. Furthermore, DANN networks trained with EG are only slightly worse on standard image classification benchmarks than baseline models, and perform much better for tasks in which task-relevant inputs are sparse. Finally, we show networks trained with EG learn heavy-tailed weight distributions that are more similar to observed biological weight distributions, and provide preliminary evidence that they perform better than GD-trained networks at higher weight-sparsity levels.