A NEURAL REPRESENTATION OF SKETCH DRAWINGS
We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on a dataset of human-drawn images representing many different classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.
This paper introduces a neural network architecture for generating sketch drawings. The authors propose that this is particularly interesting over generating pixel data as it emphasises more human concepts. I agree. The contribution of this paper of this paper is two-fold. Firstly, the paper introduces a large sketch dataset that future papers can rely on. Secondly, the paper introduces the model for generating sketch drawings.
The model is inspired by the variational autoencoder. However, the proposed method departs from the theory that justifies the variational autoencoder. I believe the following things would be interesting points to discuss / follow up:
- The paper preliminarily investigates the influence of the KL regularisation term on a validation data likelihood. It seems to have a negative impact for the range of values that are discussed. However, I would expect there to be an optimum. Does the KL term help prevent overfitting at some stage? Answering this question may help understand what influence variational inference has on this model.
- The decoder model has randomness injected in it at every stage of the RNN. Because of this, the latent state actually encodes a distribution over drawings, rather than a single drawing. It seems plausible that this is one of the reasons that the model cannot obtain a high likelihood with a high KL regularisation term. Would it help to rephrase the model to make the mapping from latent representation to drawing more deterministic? This definitely would bring it closer to the way the VAE was originally introduced.
- The unconditional generative model *only* relies on the "injected randomness" for generating drawings, as the initial state is initialised to 0. This also is not in the spirit of the original VAE, where unconditional generation involves sampling from the prior over the latent space.
I believe the design choices made by the authors to be valid in order to get things to work. But it would be interesting to see why a more straightforward application of theory perhaps *doesn't* work as well (or whether it works better). This would help interesting applications inform what is wrong with current theoretical views.
Overall, I would argue that this paper is a clear accept.