  An obstacle to artificial general intelligence is set by continual learning
of multiple tasks of different nature. Recently, various heuristic tricks, both
from machine learning and from neuroscience angles, were proposed, but they
lack a unified theory ground. Here, we focus on continual learning in
single-layered and multi-layered neural networks of binary weights. A
variational Bayesian learning setting is thus proposed, where the neural
networks are trained in a field-space, rather than gradient-ill-defined
discrete-weight space, and furthermore, weight uncertainty is naturally
incorporated, and modulates synaptic resources among tasks. From a physics
perspective, we translate the variational continual learning into Franz-Parisi
thermodynamic potential framework, where previous task knowledge acts as a
prior and a reference as well. We thus interpret the continual learning of the
binary perceptron in a teacher-student setting as a Franz-Parisi potential
computation. The learning performance can then be analytically studied with
mean-field order parameters, whose predictions coincide with numerical
experiments using stochastic gradient descent methods. Based on the variational
principle and Gaussian field approximation of internal preactivations in hidden
layers, we also derive the learning algorithm considering weight uncertainty,
which solves the continual learning with binary weights using multi-layered
neural networks, and performs better than the currently available
metaplasticity algorithm. Our proposed principled frameworks also connect to
elastic weight consolidation, weight-uncertainty modulated learning, and
neuroscience inspired metaplasticity, providing a theory-grounded method for
the real-world multi-task learning with deep networks.

Statistical Mechanics;Disordered Systems and Neural Networks;Machine Learning;Neurons and Cognition