ePoster, COSYNE 2023
Learning in neural networks with brain-inspired geometry
Jonathan Cornford, Arna Ghosh, Gauthier Gidel and Blake Richards
McGill University; McGill Univeristy; School of Computer Science; University of Montreal
- Montreal, Canada
- Machine Learning, Deep Learning and
Abstract
There are many differences between artificial and biological neural networks. However, some of those less frequently considered are the differences between their respective weight distributions and dynamics. Here, we take inspiration from neuroscience and train networks with a multiplicative update algorithm called exponentiated gradient descent (EG). In comparison to gradient descent (GD), this corresponds to changing geometries and measuring parameter differences with the un-normalised relative entropy function instead of Euclidean distance.
In line with biology, the multiplicative updates prescribed by EG are unable to change weight polarity, and as such we show a feed-forward inhibition-inspired “Dale’s ANN” (DANN) architecture learns better with EG than standard networks. Furthermore, DANN networks trained with EG are only slightly worse on standard image classification benchmarks than baseline models, and perform much better for tasks in which task-relevant inputs are sparse. Finally, we show networks trained with EG learn heavy-tailed weight distributions that are more similar to observed biological weight distributions, and provide preliminary evidence that they perform better than GD-trained networks at higher weight-sparsity levels.
Topics
- biological neural networks, search World Wide
- weight distributions, search World Wide
- exponentiated gradient descent, search World Wide
- gradient descent, search World Wide
- multiplicative updates, search World Wide
- Dale’s ANN, search World Wide
- feed-forward inhibition, search World Wide
- heavy-tailed weight distributions, search World Wide
Details
- Session
- Poster Session I