We first pose the Unsupervised Progressive Learning (UPL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes increases with time. If some limited labeled data is also available, those representations can be associated with specific classes, thus enabling classification tasks. To solve the UPL problem, we propose an architecture that involves an online clustering module, called Self-Taught Associative Memory (STAM). Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical representations rather than specific examples. The goal of this paper is to introduce the UPL problem, describe the STAM architecture, and evaluate the latter in the UPL context.
The paper presents a new problem formulation in the broader area of lifelong learning.
Specifically, the Unsupervised Progressive Learning (UPL) problem, requires a learner to consume a stream of data, where each data point is associated with a class, but only very few labels are provided. Similar to continuous learning the learner can only access current and not access previous data in the stream. The paper clearly describes how this setup is different to commonly other learning setups studied, including continual learning. 
To solve this problem the paper proposes a deep-learning-free approach based on clustering and novelty detection.
Strength:
-	I think the problem formulation is very realistic and interesting and I am not aware of it being explored previously
-	The newly proposed architecture STAM is also interesting.
-	The paper evaluates STAM on UPL on three datasets, MNIST, EMNIST, SVHN
-	The paper ablates several aspects of the model evaluates the effect of hyper parameters.
Weaknesses:
1.	The paper misses to include baselines, both, w.r.t. the proposed method, as well as for the learning problem. 
1.1.	The paper argues that deep learning cannot work, I am wondering, why don’t include standard baselines, e.g. auto-encoder based with a prototype stored whenever a labeled example is seen. (For single pass continual learning, see e.g. [B], although that work is supervised)
1.2.	Hebbian learning inspired algorithms might similarly form a valuable baseline
2.	The UPL problem makes sense to me, however, the experimental setup could be improved. Specifically, the paper uses the entire dataset to find hyperparameters. This is highly concerning as this basically means the entire dataset has been seen multiple times before the final pass over the dataset (the one with the best hyperparameters) is performed. This is basically a contradiction to the setup of UPL, which assumes all data is seen only once.
3.	Related Work:
3.1.	[A] seems to study a similar setup and proposes an approach which should also be applicable in this work.

Conclusion:
The paper’s contribution is a new learning setup (UPL) and an approach (STAM). While it remains unclear if STAM can generalize to realistic images, the idea of UPL is very interesting and speaks for accepting the paper, although there are several weaknesses the authors should address.
It would also be great if the authors plan to release the exact experimental setup (i.e. which images are sampled in which order and which ones are labeled) so future work can compare to this work.
Reference:
[A] Continuous Online Sequence Learning with an Unsupervised Neural Network Model
Yuwei Cui, Subutai Ahmad and Jeff Hawkins; Neural Computation, 2016
[B] Gradient episodic memory for continual learning, D Lopez-Paz, MA Ranzato ; NeurIPS, 2017
==== Post rebuttal comment:
Thanks for the clarifications and additional experiments;
I do think the the paper should be accepted and increased my score.
Regarding 2) I was thinking more of using a small part of the dataset to do hyper parameter selection, see e.g. 
Efficient Lifelong Learning with A-GEM
A Chaudhry, MA Ranzato, M Rohrbach, M Elhoseiny
International Conference on Learning Representations (ICLR)