Variational RNNs are proposed to output "creative" sequences. Ideally, a collection of sequences produced by a variational RNN should be of both high quality and high variety. However, existing decoders for variational RNNs suffer from a trade-off between quality and variety. In this paper, we seek to learn a variational RNN that decodes high-quality and high-variety sequences. We propose the Self-Improving Collaborative GAN (SIC-GAN), where there are two generators (variational RNNs) collaborating with each other to output a sequence and aiming to trick the discriminator into believing the sequence is of good quality. By deliberately weakening one generator, we can make another stronger in balancing quality and variety. We conduct experiments using the QuickDraw dataset and the results demonstrate the effectiveness of SIC-GAN empirically.
This paper baffles me. It appears to be a stochastic RNN with skip connections (so it's conditioned on the last two states rather than last one) trained by an adversarial objective (which is no small feat to make work for sequential tasks) with results shown on the firetruck category of the QuickDraw dataset. Yet the authors claim significantly more importance for the work than I think it merits.
First, there is nothing variational about their variational RNN. They seem to use the term to be equivalent to "stochastic", "probabilistic" or "noisy" rather than having anything to do with optimizing a variational bound. To strike the right balance between pretension and accuracy, I would suggest substituting the word "stochastic"  everywhere "variational" is used.
Second, there is nothing self-improving or collaborative about their self-improving collaborative GAN. Once the architecture is chosen to share the weights between the weak and strong generator, the only difference between the two is that the weak generator has greater noise at the output. In this sense the architecture should really be seen as a single model with different noise levels at alternating steps. In this sense, I am not entirely clear on what the difference is between the SIC-GAN and their noisy GAN baseline - presumably the only difference is that the noisy GAN is conditioned on a single timestep instead of two at a time? The claim that these models are somehow "self-improving" baffles me as well - all machine learning models are self-improving, that is the point of learning. The authors make a comparison to AlphaGo Zero's use of self-play, but here the weak and strong generators are on the same side of the game, and because there are no game rules provided beyond "reproduce the training set", there is no possibility of discovery beyond what is human-provided, contrary to the authors' claim.
Third, the total absence of mathematical notation made it hard in places to follow exactly what the models were doing. While there are plenty of papers explaining the GAN framework to a novice, at least some clear description of the baseline architectures would be appreciated (for instance, a clearer explanation of how the SIC-GAN differs from the noisy GAN). Also the description of the soft $\ell_1$ loss (which the authors call the "1-loss" for some reason) would benefit from a clearer mathematical exposition.
Fourth, the experiments seem too focused on the firetruck category of the QuickDraw dataset. As it was the only example shown, it's difficult to evaluate their claim that this is a general method for improving variety without sacrificing quality. Their chosen metrics for variety and detail are somewhat subjective, as they depend on the fact that some categories in the QuickDraw dataset resemble firetrucks in the fine detail while others resemble firetrucks in outline. This is not a generalizable metric. Human evaluation of the relative quality and variety would likely suffice.
Lastly, the entire section on the strong-weak collaborative GAN seems to add nothing. They describe an entire training regiment for the model, yet never provide any actual experimental results using that model, so the entire section seems only to motivate the SIC-GAN which, again, seems like a fairly ordinary architectural extension to GANs with RNN generators.
The results presented on QuickDraw do seem nice, and to the best of my knowledge it is the first (or at least best) applications of GANs to QuickDraw - if they refocused the paper on GAN architectures for sketching and provided more generalizable metrics of quality and variety it could be made into a good paper.