Nonlinear Hebbian plasticity for dimensionality reduction

Brains often infer low-dimensional features from high-dimensional input. For example, during visual processing a picture captured by the retina is processed hierarchically through the ventral stream, which gradually infers higher-level features encoded by the lower number of neurons. However, it is still unclear how neural networks learn to perform such tasks. Hebbian Learning (HL), a local form of plasticity that uses pre- and postsynaptic activity to change synaptic efficacy, has been used extensively in receptive field models, sparse coding, independent and principal component analysis, and has also been proposed for learning dimensionality reduction. However, HL is traditionally used to only change synaptic weights, but no cellular parameters, such as, e.g. the activation function of a neuron. What's more, using a linear activation function, HL performs poorly for dimensionality reduction in data with intricate nonlinear structure. Here we show that using an optimized activation function allows one to account for nonlinear dependencies in the input, and making dimensionality reduction of the output possible. We demonstrate how such "nonlinear" HL can learn dimensionality reduction in a multi-layer feed-forward network, by considering neural activations as functions with learnable parameters. To find the optimal combination of parameters, we minimize a loss function w.r.t to the parameters via gradient descent. This loss function is derived from the UMAP manifold learning algorithm, and indicates how well the original high-dimensional data structure is preserved in a low-dimensional output of the network. Our approach allows adjusting neural activations in such a way that the network will learn how to reduce data dimensionality using Hebbian plasticity, demonstrating that we may not want to think of learning as a purely synaptic process.