Topic: exponentiated gradient descent

There are many differences between artificial and biological neural networks. However, some of those less frequently considered are the differences between their respective weight distributions and dynamics. Here, we take inspiration from neuroscience and train networks with a multiplicative update algorithm called exponentiated gradient descent (EG). In comparison to gradient descent (GD), this corresponds to changing geometries and measuring parameter differences with the un-normalised relative entropy function instead of Euclidean distance. In line with biology, the multiplicative updates p