One of the most surprising and exciting discoveries in supervising learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it has been observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). In this paper, we perform an exhaustive study of different aspects of overparameterization in unsupervised learning via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out loglikelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that, when learning using methods such as variational inference, larger models can significantly increase the number of ground truth latent variables recovered.
The paper “aims to be a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings. ” The author’s empirical study is comprehensive, and to my knowledge the most detailed published work on this to date. Specifically, the authors empirically study 
- the ability of networks to recover latent variables
- the effects of extreme overparameterization
- the effects the training method (e.g. batch size)
- latent variable stability over the course of training
In line with the findings for supervised settings, the authors find that overparameterization is often beneficial, and that overfitting is a surprisingly small issue. This is an interesting and useful observation, particularly since it at first sight appears to be in disagreement with some earlier work (the authors suggest explanations for the differing observations). 
As the authors point out (and I agree), the paper constitutes a compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods. 
The authors perform studies on a range of different real-world and synthetic datasets. 
The paper is well-written, well-structured, and easy to follow. Relevant literature has been cited. The appendices contain a wealth of details that will make this work reproducible. 
Decision: weak accept. The paper contains some new insights, but its contributions are not quite as substantial (e.g. lack of precise mathematical statements) or surprising as those in stronger ICLR papers. 
A small gripe: the authors promise “ a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings”. I would argue that “making precise” is too strong for what the paper actually delivers. I suggest rewording this.