Generative replay (GR) is a method to alleviate catastrophic forgetting in continual learning (CL) by generating previous task data and learning them together with the data from new tasks. In this paper, we propose discriminative variational autoencoder (DiVA) to address the GR-based CL problem. DiVA has class-wise discriminative latent embeddings by maximizing the mutual information between classes and latent variables of VAE. Thus, DiVA is directly applicable to classification and class-conditional generation, which are efficient and effective properties in the GR-based CL scenario. Furthermore, we use a novel trick based on domain translation to cover natural images which are challenging to GR-based methods. As a result, DiVA achieved competitive or higher accuracy compared to state-of-the-art algorithms in Permuted MNIST, Split MNIST, and Split CIFAR10 settings.
-- This paper seeks to combine several ideas together to propose an approach for image classification based continual learning tasks. In this effort, the paper combines previously published approaches from generative modeling with VAEs, mutual information regularization and domain adaptation. 
I am a making a recommendation for reject for this paper with the main reason being that I believe the primary derivations for their method appear flawed. 
--In the main section describing the approach (Section 4), the authors start with a claim that Equation 1 and 2 are equal; I don’t believe 1 and 2 are equal.
--In Section 4.1, it appears that they are instead making a claim about Equation 2 being a bound for equation 1; but even this derivation appears to have a problem. The following is the concern:
--In the second line of Equation 5, the KL term appears to be measuring a distance between distributions on two different variables; z|c and c|z. If one were to interpret the second one as the unnormalized distribution on z defined via the likelihood for c given z; even this has an issue because then the expression for KL where we plug the unnormalized density in place of the normalized need not be positive which is something they need to derive their bound.
--Another issue is that the regularization lambda should apply to both the terms in the bound but in Equation (7) only appears selectively for one of the two terms. 
It is also not clear how the loss function proposed differs from that of the CDVAE, etc.  If the novelty is in applying to continual learning and new datasets, it is not clear that this is sufficient.
Additional feedback for authors (not part of the main decision reasoning):
- What is dt in Algorithm 1 description?
Figure 1:
-typo “implmented”
-What’s the 3d plot supposed to represent?
Doesn't the classification loss have a dependency on the input condition?
--What does a "heavy classifier" imply concretely? 
--“Redundant weights” seems like not a very strong constraint especially for a small cardinality label space (like 10, in the case of this paper).
--The notation for the proposed parameters theta, theta’, phi, phi’ are not consistent with the notation in the intro section, where phi was used for the encoder and theta for the decoder. In later sections they use theta and theta’ for encoder/decoder resp.
-- “When the encoder and decoder networks are sufficiently complex, it is enough to implement each the prior and classification network as one fully-connected layer” → what do the authors mean “when … networks are sufficiently complex” or do they actually mean when the “when the problem is simple enough”?