  Catastrophic forgetting has been a significant problem hindering the
deployment of deep learning algorithms in the continual learning setting.
Numerous methods have been proposed to address the catastrophic forgetting
problem where an agent loses its generalization power of old tasks while
learning new tasks. We put forward an alternative strategy to handle the
catastrophic forgetting with knowledge amalgamation (CFA), which learns a
student network from multiple heterogeneous teacher models specializing in
previous tasks and can be applied to current offline methods. The knowledge
amalgamation process is carried out in a single-head manner with only a
selected number of memorized samples and no annotations. The teachers and
students do not need to share the same network structure, allowing
heterogeneous tasks to be adapted to a compact or sparse data representation.
We compare our method with competitive baselines from different strategies,
demonstrating our approach's advantages.

Machine Learning;Artificial Intelligence