META-LEARNING UPDATE RULES FOR UNSUPER- VISED REPRESENTATION LEARNING
A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithman unsupervised weight update rule -that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.
This paper introduces a novel meta-learning approach to unsupervised representation learning where an update rule for a base model (i.e., an MLP) is meta-learned using a supervised meta-objective (i.e., a few-shot linear regression from the learned representation to classification GTs). Unlike previous approaches, it meta-learns an update rule by directly optimizing the utility of the unsupervised representation using the meta-objective. In the phase of unsupervised representation learning, the learned update rule is used for optimizing a base model without using any other base model objective. Experimental evaluations on few-shot classification demonstrate its generalization performance over different base architectures, datasets, and even domains.  
+  Novel and interesting formulation of meta-learning by learning an unsupervised update rule for representation learning. 
+  Technically sound, and well organized overall with details documented in appendixes. 
+  Clearly written overall with helpful schematic illustrations and, in particular, a good survey of related work. 
+ Good generalization performance over different (larger and deeper) base models, activation functions, datasets, and even a different modality (text classification).
-  Motivations are not very clear in some parts. E.g., the reason for learning backward weights (V), and the choice of meta-objective.  
- Experimental evaluation is limited to few-shot classification, which is very close to the meta-learning objective used in this paper. 
- The result of text classification is interesting, but not so informative given no further analysis. E.g., why domain mismatch does not occur in this case?
I enjoyed reading this paper, and happy to recommend it as a clear accept paper. The idea of meta-learning update networks looks a promising direction worth exploring, indeed. 
I hope the authors to clarify the things I mentioned above. Experimental results are enough considering the space limit, but not great. Since the current evaluation task is quite similar to the meta-objective, evaluations on more diverse tasks would strengthen this paper. 
Finally, this paper aims at unsupervised representation learning, but it’s not clear from the current title, which is somewhat misleading. I think that's quite an important feature of this paper, so I highly recommend the authors to consider a more informative title, e.g., `Learning Rules for Unsupervised Representation Learning’ or else.