We introduce a unified probabilistic approach for deep continual learning with open set recognition, based on variational Bayesian inference. Our model combines a joint probabilistic encoder with a generative model and a linear classifier that get shared across tasks. The proposed open set recognition bounds the approximate posterior by fitting regions of high density on the basis of correctly classified data points and balances open set detection with recognition errors. Catastrophic forgetting is significantly alleviated through generative replay, where the open set recognition is used to reject statistical outliers from low density areas of the class specific posterior. Our approach naturally allows for forward and backward transfer while maintaining past knowledge without the necessity of storing old data, regularization or inferring task labels. We demonstrate compelling results in the challenging continual learning scenario of incrementally expanding the singlehead classifier for both class incremental visual and audio classification tasks, as well as incremental learning of datasets across modalities. We further quantitatively demonstrate our models ability to successfully distinguish unseen unknown data from trained known tasks and subsequently prevent misclassification.
Summary: This paper proposes a unified model for continual learning and aims to address the following problems:
Out-of-train-domain dataset recognition
Catastrophic forgetting
The out-of-domain or open set recognition model is not only used to detect outliers but also for sampling “representative data” of previous tasks for forward (and backward) transfer. 
COMMENT(S)/QUERIES:
------------------------------
 
While the experiments do justice to the contributions mentioned in the paper, I believe that certain sections need clarifications and expansion (while some can be dismissed). 
1. Given my limited knowledge in the literature on this topic, I would have appreciated a proper related work section. 
2. I believe equation (1) could have been moved to supplementary for reference since equation (2) is the only important equation in the paper. 
I don’t believe Equation (1) is really used for testing purposes. It’s just that the data is sent through the probabilistic encoder and then classified. There is no need for reconstruction of the data point.
3. It’s difficult to follow the flow of the method. Shouldn’t generative replay (algorithm 3) be placed before open set recognition of unknown and uncertain inputs (algorithm 2), since the latter is probably just used during the test, while the former affects the training procedure directly (implicit data augmentation)?
4. Major concern: It’s somewhat strange to observe that the VAE model is able to generate multiclass data so fluidly with a simple gaussian prior. This kind of challenges the belief that current generative models are unable to capture all modes perfectly. A small note about why multimodal prior was not used (which intuitively and mathematically makes more sense) and also a statement about the average time to generate multiclass multi-modal data using algorithm 3 would have been nice. 
5. In section 3.2, “For our single-head expanding classifier this ensures..”. While having a single-head expanding classifier is listed as one of the important contributions, it hasn’t been given enough justice w.r.t to the implementation details. Is it like adding a completely new classifier during the training of the new task?
The entire section on hyper-parameters could’ve been moved to the supplementary if space was a major constraint but compromising on details about an important contribution only weakens the paper. 
I especially like figure 2 in the experiments, where the importance of Weibull CDF outlier rejection prior is highlighted.
Typo: 
incorrect opening inverted comma for the word background in the introduction section (page 1)
OVERALL COMMENT
-----------------------------
The paper combines the generative, and discriminative models into one framework for multiple important tasks. While the contributions are clear in the introduction, the presentation of the paper is somewhat too complicated at a couple of places. It does not do full justice to explaining its more vital components and some parts of the method section feel like a jigsaw puzzle, where readers are heavily expected to “figure out on their own”. 
With proper presentation though, I can envision this paper contributing positively to unified frameworks in general.
Due to the above reasons. I am giving it a score of 6.