Implicit generative models are difficult to train as no explicit probability density functions are defined. Generative adversarial nets (GANs) propose a minimax framework to train such models, which suffer from mode collapse in practice due to the nature of the JS-divergence. In contrast, we propose a learning by teaching (LBT) framework to learn implicit models, which intrinsically avoid the mode collapse problem because of using the KL-divergence. In LBT, an auxiliary explicit model is introduced to learn the distribution defined by the implicit model while the later one's goal is to teach the explicit model to match the data distribution. LBT is formulated as a bilevel optimization problem, whose optimum implies that we obtain the maximum likelihood estimation of the implicit model. We adopt an unrolling approach to solve the challenging learning problem. Experimental results demonstrate the effectiveness of our method.
This work introduces a framework for learning implicit models that is robust to mode collapse. It consists in learning an explicit model of the implicit model through maximum likelihood while the later is used to teach the explicit model to better match the data distribution. The resulting bi-level optimization is carried out with truncated unrolled stochastic gradient descent.
# Quality
The method combines an interesting set of ideas. It is validated on some reasonable experiments. 
However after reading the paper, I remain with too many unanswered questions:
- Why should the method avoid mode collapse? Experiments clearly show that it indeed is resilient to mode collapse, but I have would have been curious in seeing some more discussion regarding this point. What is the exact mechanism that solves the issue?
- What is the effect of K? Is mode collapse solved only because of the unrolled gradients?
- What is the effect of M? How does the method behave for M=1, as usually done in GANs?
- What if the explicit model has not enough capacity?
- The original Unrolled GAN paper presents better results for the ring problem. Why are results worse in the experiments?
More fundamentally what is the main benefit of this approach with respect to models that can be trained straight with maximum likelihood? (e.g., flow-based neural generative models; and as required for the explicit model) Is it only to produce generative models that are fast (because they are implicit)? Why not training only the explicit model directly on the data?
# Clarity
The paper is in general well-written, although some elements could be removed to actually help with the presentation.
- The development around influence functions could be removed, as the method ends up instead making use of truncated unrolled gradients.
- The theoretical analysis is straightforward and could be compressed in a single paragraph to motivate the method.
# Originality
The method makes use of several ideas that have been floating around and proposed in different papers. As far as I know, the combination proposed in this work is original.
# Significance
Results show clear resistance to mode collapse, which is an improvement for implicit models. However, other types of generative models generally do not suffer from this issue. Significance is therefore limited.