Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudosamples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/xxx.
Summary:
The paper proposes to use the same language model to learn multiple tasks and also to generate pseudo-samples for these tasks which could be used for rehearsal while learning new tasks. The authors demonstrate that this idea works well compared to other SOTA lifelong learning methods for learning various NLP tasks using a single model.
My comments:
1. Please change the title! Language modeling is NOT all you need for lifelong language learning. Also, not every NLP task is a QA task. I do not want more papers to over-trivialize NLP by following Bryan McCann and Socher, 2018. I will not increase my scores until the title is changed.
2. A relevant model architecture based method is Sodhani et al. 2018 (Towards Training Recurrent Neural Networks for Lifelong Learning) who use Net2Net to do zero-shot expansion of the model parameters.
3. Section 3.2 - you mention that any pseudo-example which does not have only one ANS token is discarded. Can you comment on how much discarding is needed to generate the required number of pseudo-samples?
4. Why is it that every task was trained only for 9 epochs?
5. On page 5, you mention k=20. What is k? Where is this introduced?
6. On page 5, you mention that MTL is used to determine whether forgetting is caused by a lack of model capacity. I am not sure if it is correct. Can you explain?
7. Why not compare the approach with models like GEM? Keeping very few examples is ok. Even though you don’t beat GEM, it is good to see the comparison.
8. Page 7: Is there any reason why you choose to go from large to small tasks? I feel like this is a favorable order. I would like to see how the model performs if you do the reverse order.
9. Please remove the last line.
10. I assume that the authors will release the code upon acceptance of the paper.
Minor comments:
1. Page 2, 4th contribution: check the spelling for “pseudo-samples”
2. Page 2, 5th last line: “After a completing a task” - fix it.
3. Table 1: I think the description is not correct. 1fEM is for wikiSQL, not WOZ. Also, it is better if you can describe these metrics in detail in the appendix.
==================================
After rebuttal:
I am happy with the authors' response and name change. I am increasing my score.