Topic: CIFAR10

The natural world is full of noise, but the brain’s capacity for information transmission is severely limited. Therefore, discarding irrelevant information contained in sensory inputs while retaining salient features that are related to the input label, is key to survival. What are the salient features? And what are the underlying feature selection mechanisms? It is thought that the brain may implement information bottlenecks, which aim to optimise the trade-off between compression and preservation of salient information. However, information bottlenecks are notoriously difficult to implemen

Networks composed of Non-Negative Matrix Factorization (NNMF) [1] modules can serve as abstractions of real neural networks. In particular, NNMF networks can be extended easily to perform computations based on stochastic spikes [2,3]. However, due to training via Autograd approaches following exact gradients to train deep networks of NNMF modules currently requires prohibitively large amounts of memory and is very slow. Here we present an approximative backpropagation (BP) method for optimizing deep NNMF networks. NNMF layers have latent variables that are updated iteratively towards a fixed

Our recent work presents a stochastic process model of the activations within an ANN and shows a promising indicator to distinguish memorizing from generalizing ANNs. The average λ, or mean firing rate (MFR), of a hidden layer, shows stable differences between memorizing and generalizing networks, comparatively independent of the underlying data used for evaluation. We first show the performance of this indicator during training on benchmark computer vision datasets such as MNIST and CIFAR-10. In a second step, we extend the work to the real-life use case of calibrating a pre-trained model to