Probability distillation has recently been of interest to deep learning practitioners as it presents a practical solution for sampling from autoregressive models for deployment in real-time applications. We identify a pathological optimization issue with the commonly adopted stochastic minimization of the (reverse) KL divergence, owing to skewed gradient distribution due to curse of dimensionality. We also explore alternative principles for distillation, and show that one can achieve qualitatively better results than with KL minimization.
This paper proposes new methods for distilling a feed-forward generative model (student) from an autoregressive generative model (teacher) as an alternative to the reverse-KL divergence. The first part of the paper analyses optimization issues with the reverse KL divergence while in the second part of the paper alternatives are proposed (x-reconstruction and z-reconstruction).
Detailed comments:
1.
In abstract and other places: "sparse gradient signal from the teacher".
Sparsity implies that many of the values are exactly zero, while Section 3.1 seems to imply that some of the values might be small (or pointing towards the origin).
2.
In Section 3.1 and 3.2 the authors discuss a potential failure mode of the reverse KL:
But, proposition 1 boils down to the fact that if the student's mass is more spread out than the teacher is some direction, that it should shrink that mass closer to zero as well.
In the example of the paper: if an eigenvalue of T is smaller than 1, it would mean that the student which is spherical Gaussian, would adjust its probability mass to also be smaller in that eigenvector's direction.
As training progresses, the students mass would be much closer to the teacher and the probability of 'pointing away' from the origin would be about as likely as pointing towards.
So it's not clear at all that the described property is problematic for optimization, as it could as well be interpreted as the student trying to fit the teacher's distribution better.
3.
Was the KL between P_S(x_i | z_<i) and P_T(x_i | x_<i) computed analytically? If these conditional distributions are Gaussian (which they are in many of the examples) this should be trivial.
4.
Section 4 about the neural vocoder needs to be expanded: many details are missing here and although it's one of the more important experiments in the paper it's relatively neglected compared to the other parts of the paper.
5.
In the Section 4: the experiment with reverse-KL is a straw man comparison: For audio the reverse KL was only proposed in combination with the power loss (Oord et al). Two additional experiments would make the result a lot stronger: KL+power-loss and X-recon+power-loss. Because if the x-recon method does not work well together with the power-loss, its practical applicability seems limited.
The proposed methods are interesting, because they are elegant and seems to work reasonably well on the tasks tried. The first part of the paper about gradient sparsity/orientation needs to be addressed. Section 4 should be expanded and an additional comparison should be made.
I would change my rating if these issues were addressed.