Which generative model is the most suitable for Continual Learning? This paper aims at evaluating and comparing generative models on disjoint sequential image generation tasks. We investigate how several models learn and forget, considering various strategies: rehearsal, regularization, generative replay and fine-tuning. We used two quantitative metrics to estimate the generation quality and memory ability. We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10). We found that among all models, the original GAN performs best and among Continual Learning strategies, generative replay outperforms all other methods. Even if we found satisfactory combinations on MNIST and Fashion MNIST, training generative models sequentially on CIFAR10 is particularly instable, and remains a challenge. Our code is anonymously available online 1 .
This paper performs an empirical comparison of models and CL methods in a generative setting. The main motivation of the paper is to make statements about which model/method combinations are best to use for generative tasks in the CL setting. In short, the paper provides an empirical analysis and evaluation of the combination of CL methods and generative models.
The datasets used for comparison are MNIST, Fashion MNIST, and CIFAR10. For each dataset, sequential (class by class) generative tasks are introduced, aligning with the CL setting. The models investigated are VAEs, GANs, and WGANs, along with their (class) conditional counter-parts. The CL methods investigated are (i) fine-tuning (a simple baseline), (ii) rehearsal methods, (iii) elastic weight consolidation (EWC), and (iv) generative replay (GR). The authors propose to use two evaluation metrics: Fréchet Inception Distance (FID) measures the quality of the generated images, and fitting capacity (FC) measures the usefulness of the images to train classifiers.
Pros:
- The authors are correct in pointing out that most of the work on CL has been restricted to the discriminative case, and that there is value in exploring generative tasks in the CL setting.
- Empirical and experimental evaluation of this sort are useful, and help the community better understand the relationship between model, CL method, and task. Such an evaluation and in-depth analysis is welcomed in CL, especially in the generative setting.
- The authors draw a number of useful conclusions e.g., regarding the usefulness and dangers of employing the different CL methods.
Cons:
- My main concern with this paper regards the evaluation metrics used. The authors propose quality metrics for the generative model, both of which (directly or indirectly) measure the quality of the generated images. In this setting, it is unsurprising that GANs outperform VAEs, as they are known to generate higher-quality images. This however, does not necessarily mean that they are better at the continual learning task (i.e., avoiding catastrophic forgetting). It seems to me that one source from which to draw would be [1], which conducted a very rigorous and useful empirical evaluation of generative models, and the methodology followed there (i.e., evaluating marginal log-likelihoods via annealed importance sampling) would be more convincing evidence for empirical comparison of models, as it would somewhat detach the quality of the generated images from the ability of the model to avoid catastrophic forgetting.
Using their proposed image-quality metrics, the authors make statements such as: "Our results do not give a clear distinction between conditional and unconditional models. However, adversarial methods perform significantly better than variational methods. GANs variants are able to produce better, sharper quality and variety of samples, as observed in Fig. 13 and 14 in Appendix G. Hence, adversarial methods seem more viable for CL." My impression is that this statement on the viability of VAEs vs GANs for CL, which is a major point of the paper, does not follow from the empirical results on the quality of the generated images. It seems quite predictable that the GAN-based models would produce higher quality images, regardless of catastrophic forgetting.
Additional (minor) comments:
- Sec. 2 could consist of a more thorough review of the literature, with a more in-depth comparison of the different CL methods proposed and evaluated in the paper.
- Sec. 2 contains a number of statements of the form: "restricted to VAEs only". For many of the cases it is not immediately clear why this is true, and in my opinion the authors should either drop those comments, or make them rigorous.
- VCL "use specific weights for each task, which only works for the setting where the number of tasks is known in advance". Unclear what exactly this means or why this is true.
- "while the teacher retains knowledge" - how does it "retain knowledge", how is this then transferred to the student, and why is this restricted to VAEs?
Experimental protocol:
- Core-sets for the rehearsal as proposed by [1] could be an interesting extension. It is unclear how the samples were selected for rehearsal, and core-sets represent a principled way to do so, that would also be interesting to compare in this setting to a random baseline.
- For VAEs, a potentially better metric of their ability (other than the log-likelihood as suggested by [2]) would be fitting capacity (or other metric) over learned latent space rather than the reconstructed image-space.
Overall, my impression is that while an empirical analysis of CL methods in the generative setting is a useful concept, the submission in its current form requires some improvement. In particular, I am worried that the choice of evaluation metrics may lead to incorrect (or partially correct) conclusions, which could of course have a negative impact on the research into CL. It also seems that the paper could use some further polishing in both writing and presentation. As such, I encourage the authors to continue the work on this empirical analysis, and perhaps submit in again to future conferences.
[1] - Nguyen et al. Variational Continual Learning, ICLR 2018
[2] - Wu et al. On the Quantitative Analysis of Decoder-Based Generative Models, ICLR 2017