title: Continual Learning in Deep Networks: an Analysis of the Last Layer
abstract: We study how different output layers in a  deep neural network learn and forget in continual learning settings. The following three  factors  can affect catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layers may address (1) and (2).  Some potential solutions to those issues are proposed and evaluated here in  several continual learning scenarios. We show that the best-performing type of the output layer  depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, it turns out that changing  parameterization is sufficient in order to  achieve a significantly better performance, whithout introducing a continual-learning algorithm and instead using the standard SGD to train a model.  Our analysis and results  shed light on the dynamics of the output layer in continual learning scenarios,  and suggest a way of  selecting the best type of  output layer for a given scenario.
authors: Timothee LESORT Thomas George Irina Rish
hIndex: Irina Rish 33