Dissecting the Factors of Metaplasticity with Meta-Continual Learning

Metaplasticity is the change in plasticity of a synapse, and an important mechanism for resolving the stability-plasticity dilemma. However, the exact synaptic or neuron properties that modulate metaplasticity still remains unclear.

In this work we use meta-continual learning to discover the importance of factors that contribute to metaplasticity. We use a linear model to assign the relative contribution of four commonly used synaptic properties to metaplasticity: the Hessian, gradient, magnitude of the weights, and activity of the post-synaptic neuron. The coefficients of the linear model are meta-optimized jointly with the neural network on multiple tasks of the Om?niglot dataset continuously with a retained accuracy meta-objective.

We find that the weight and activity make the most significant contributions to metaplasticity, while the Hessian and gradient are unimportant. We also find that metaplasticity is required to overcome catastrophic forgetting, as opposed to fixed plasticity. It has superior retained accuracy than other continual learning methods, with at most 39.3% improvements. These results suggests that simple activity-based mechanisms of meta?plasticity may be sufficient and cast doubt over the relevance of gradient-based and Hessian-based ones.