LEARNING TO LEARN WITHOUT FORGETTING BY MAXIMIZING TRANSFER AND MINIMIZING INTERFERENCE
Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. 1  We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.
The transfer/ interference perspective of lifelong learning is well motivated, and combining the meta-learning literature with the continual learning literature (applying reptile twice), even if seems obvious, wasn't explored before. In addition, this paper shows that a lot of gain can be obtained if one uses more randomized and representative memory (reservoir sampling). However, I'm not entirely convinced with the technical contributions and the analysis provided to support the claims in the paper, good enough for me to accept it in its current form. Please find below my concerns and I'm more than happy to change my mind if the answers are convincing.
Main concerns:
1) The trade-off between transfer and interference, which is one of the main contributions of this paper, has recently been pointed out by [1,2]. GEM[1] talks about it in terms of forward transfer and RWalk[2] in terms of "intransigence". Please clarify how "transfer" is different from these. A clear distinction will strengthen the contribution, otherwise, it seems like the paper talks about the same concepts with different terminologies, which will increase confusion in the literature.    
2) Provide intuitions about equations (1) and (2). Also, why is this assumption correct in the case of "incremental learning" where the loss surface itself is changing for new tasks?
3) The paper mentions that the performance for the current task isn't an issue, which to me isn't that obvious as if the evaluation setting is "single-head [2]" then the performance on current task becomes an issue as we move forwards over tasks because of the rigidity of the network to learn new tasks. Please clarify.
4) In eq (4), the second sample (j) is also from the same dataset for which the loss is being minimized. Intuitively it makes sense to not to optimize loss for L(xj, yj) in order to enforce transfer. Please clarify.
5) Since the claim is to improve the "transfer-interference" trade-off, how can we verify this just using accuracy? Any metric to quantify these? What about forgetting and forward transfer measures as discussed in [1,2]. Without these, its hard to say what exactly the algorithm is buying.
6) Why there isn't any result showing MER without reservoir sampling. Also, please comment on the computational efficiency of the method (which is crucial for online learning), as it seems to be very slow. 
7)The supervised learning experiments are only shown on the MNIST. Maybe, at least show on CONV-NET/ RESNET (CIFAR etc).
8) It is not clear from where the gains are coming. Do the ablation where instead of using two loops of reptile you use one loop.
Minor:
=======
1) In the abstract, please clarify what you mean by "future gradient". Is it gradient over "unseen" task, or "unseen" data point of the same task. It's clear after reading the manuscript, but takes a while to reach that stage.
2) Please clarify the difference between stationary and non-stationary distribution, or at least cite a paper with the proper definition.
3) Please define the problem precisely. Like a mathematical problem definition is missing which makes it hard to follow the paper. Clarify the evaluation setting (multi/single head etc [2])
4) No citation provided for "reservoir sampling" which is an important ingredient of this entire algorithm.
5) Please mention appendix sections as well when referred to appendix.
6) Provide citations for "meta-learning" in section 1.
[1] GEM: Gradient episodic memory for continual learning, NIPS17.
[2] RWalk: Riemannian walk for incremental learning: Understanding forgetting and intransigence, ECCV2018.