Published as a conference paper at ICLR 2018 OVERCOMING CATASTROPHIC INTERFERENCE USING CONCEPTOR-AIDED BACKPROPAGATION
Catastrophic interference has been a major roadblock in the research of continual learning. Here we propose a variant of the back-propagation algorithm, "conceptor-aided backprop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks. Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting. CAB extends these results to deep feedforward networks. On the disjoint and permuted MNIST tasks, CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed.
[Reviewed on January 12th]
This article applies the notion of “conceptors” -- a form of regulariser introduced by the same author a few years ago, exhibiting appealing boolean logic pseudo-operations -- to prevent forgetting in continual learning,more precisely in the training of neural networks on sequential tasks. It proposes itself as an improvement over the main recent development of the field, namely Elastic Weight Consolidation.  After a brief and clear introduction to conceptors and their application to ridge regression, the authors explain how to inject conceptors into Stochastic Gradient Descent and finally, the real innovation of the paper, into Backpropagation. Follows a section of experiments on variants of MNIST commonly used for continual learning.
Continual learning in neural networks is a hot topic, and this article contributes a very interesting idea. The notion of conceptors is appealing in this particular use for its interpretation in terms of regularizer and in terms of Boolean logic.  The numeric examples, although quite toy, provide a clear illustration.
A few things are still missing to back the strong claims of this paper:
* Some considerations of the computational costs: the reliance on the full NxN correlation matrix R makes me fear it might be costly, as it is applied to every layer of the neural networks and hence is the largest number of units in a layer.  This is of course much lighter than if it were the covariance matrix of all the weights, which would be daunting, but still deserves to be addressed, if only with wall time measures.
* It could also be welcome to use a more grounded vocabulary, e.g. on p.2 “Figure 1 shows examples of conceptors computer from three clouds of sample state points coming from a hypothetical 3-neuron recurrent network that was drive with input signals from three difference sources” could be much more simply said as “Figure 1 shows the ellipses corresponding to three sets of R^3 points”. Being less grandiose would make the value of this article nicely on its own.
* Some examples beyond the contrived MNIST toy examples would be welcome. For example, the main method this article is compared to (EWC) had a very strong section on Reinforcement learning examples in the Atari framework, not only as an illustration but also as a motivation. I realise not everyone has the computational or engineering resources to try extensively on multiple benchmarks from classification to reinforcement learning. Nevertheless, without going to that extreme, it might be worth adding an extra demo on something bigger than MNIST. The authors transparently explain in their answer that they do not (yet!) belong to the deep learning community and hope finding some collaborations to pursue this further. If I may make a suggestion, I think their work would get much stronger impact by  doing it the reverse way: first finding the collaboration, then adding this extra empirical results, which then leads to a bigger impact publication.
The later point would normally make me attribute a score of "6: Marginally above acceptance threshold" by current DL community standards, but because there is such a pressing need for methods to tackle this problem, and because this article can generate thinking along new lines about this, I give it a 7 : Good paper, accept.