Universal Style Transfer via Feature Transforms
Universal style transfer aims to transfer arbitrary visual styles to content images. Existing feed-forward based methods, while enjoying the inference efficiency, are mainly limited by inability of generalizing to unseen styles or compromised visual quality. In this paper, we present a simple yet effective method that tackles these limitations without training on any pre-defined styles. The key ingredient of our method is a pair of feature transforms, whitening and coloring, that are embedded to an image reconstruction network. The whitening and coloring transforms reflect a direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer. We demonstrate the effectiveness of our algorithm by generating high-quality stylized images with comparisons to a number of recent methods. We also analyze our method by visualizing the whitened features and synthesizing textures via simple feature coloring.
This submission describes a model for general and efficient style transfer. It is general in the sense that it does not need training for each style image that one would want to apply (unlike [26]), and it is efficient because it does not require solving an optimization problem as the initial work of Gatsby et al. [9].
The main idea is to train a deep decoder network, so that from any given layer in a VGG, one can generate the most likely image. This decoder is not trained as a classical autoencoder. Instead, given a fixed supervised VGG, an decoder counterpart of a VGG is trained so as to reconstruct the image. 
Given this trained decoder, the authors propose to transfer style by taking a given layer, make the covariance of the activations the identity and in a second step, make them follow the covariance of the style image at the same layer (called coloring). Given these new activations the desired style-transfered image is obtained by applying the corresponding decoder. 
Pros:
- The proposed approach is very simple and intuitive. It does not require fancy losses, and uses simple baseline techniques to approximately match the distributions of activations.
- This paper exhibits the very large expressive power of CNNs. Given a well trained decoder, simple linear operations in the feature space give meaningful outputs.
- The proposed Coarse to Fine approach leads to very good visual performance.
- The proposed algorithm is fairly efficient as for each new stye, it only requires computing several singular value decompositions (two for each layer, one for the image, one for the style).
- The proposed model only has one parameter, namely the tradeoff between style and content. This is a great advantage to say [26].
- Overall the paper reads well and is easy to follow.
Cons:
- The description of the Decoder training could be made more clear by introducing the fixed encoder, let's say \phi. Then the training of the decoder could be written as the minimization w.r.t. a decoder \psi of the loss. That would avoid using strange notations including I_\input I_\output (which actually depends on the function that is optimized over...).
- The formal description of the whitening and coloring could be made more practical by introducing the SVD of the activations (instead of using the eigenvalue decomposition of the covariance matrix).
Questions:
- To a large extent, this approach relies on the training of a good decoder, they quality of which strongly depends on the choice of data used for training. It would be interesting to see more results with worse decoders, trained on less data (or images with different statistics). 
- The model assumes that the "support" of the decoder (subset of the input space that was covered by the inputs) contains the activations of the style images. Provided that this is true and that the decoder is well regularized, the output should be correct. Would it be possible to assess to what degree does a dataset such as COCO cover well this space? Otherwise, the decoder might have ill-defined behaviors for parts of the input space that it has never seen during training.
Overall, I think this submission in interesting and shows good results with a simple approach. I lean towards acceptance.