Artistic style transfer is the problem of synthesizing an image with content similar to a given image and style similar to another. Although recent feed-forward neural networks can generate stylized images in real-time, these models produce a single stylization given a pair of style/content images, and the user doesn't have control over the synthesized output. Moreover, the style transfer depends on the hyper-parameters of the model with varying "optimum" for different input images. Therefore, if the stylized output is not appealing to the user, she/he has to try multiple models or retrain one with different hyper-parameters to get a favorite stylization. In this paper, we address these issues by proposing a novel method which allows adjustment of crucial hyper-parameters, after the training and in real-time, through a set of manually adjustable parameters. These parameters enable the user to modify the synthesized outputs from the same pair of style/content images, in search of a favorite stylized image. Our quantitative and qualitative experiments indicate how adjusting these parameters is comparable to retraining the model with different hyper-parameters. We also demonstrate how these parameters can be randomized to generate results which are diverse but still very similar in style and content.
The paper presents an approach for style transfer with controlable parameters. The controllable parameters correspond to the weights associated to "style losses" or ordinary style transfer models (distance between gram matrices of generated vs style image at specific layers of a network). The authors propose to learn a single architecture that takes these parameters as input to generate an image that resembles what would be generated by optimizing directly on these parameters. Examples of transfer and of the effect of these parameters are given. A quantitative evaluation shows that the effect of changing the parameters of the new network has the effect of reducing the loss at the desired layers.
Overall, I found the idea of learning controllable parameters interesting. The technical contribution is to show that learning the weights of instance normalization as a function of the hyperparameters actually is satisfactory. To be honest, I am not sure to understand why playing on instance normalization weights turns to be effective for learning an equivalent of a reduction of per-layer loss. I would have liked more motivation and intuition behind it.
The paper seems overall a bit incremental with respect to prior work on style transfer. While adjustable parameters could have application in more general tasks of image generation (for instance, in the line of work on disentangled representations), it is unclear how the specific approach of learning instance normalization weights extends beyond style transfer. As such, my feeling is that the paper is borderline.
other comments:
- I found Figure 7 important because it guarantees learning has the desired effect. However, there is a bit a lack of baseline/topline: how do the true losses decrease when their weights increase?