A Powerful Generative Model Using Random Weights for the Deep Image Representation
To what extent is the success of deep visualization due to the training? Could we do deep visualization using untrained, random weight networks? To address this issue, we explore new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks. First we invert representations in feature spaces and reconstruct images from white noise inputs. The reconstruction quality is statistically higher than that of the same method applied on well trained networks with the same architecture. Next we synthesize textures using scaled correlations of representations in multiple layers and our results are almost indistinguishable with the original natural texture and the synthesized textures based on the trained network. Third, by recasting the content of an image in the style of various artworks, we create artistic images with high perceptual quality, highly competitive to the prior work of Gatys et al. on pretrained networks. To our knowledge this is the first demonstration of image representations using untrained deep neural networks. Our work provides a new and fascinating tool to study the representation of deep network architecture and sheds light on new understandings on deep visualization. It may possibly lead to a way to compare network architectures without training.
This paper performs 3 optimization tasks that are popular in deep visualization: 1) feature inversion, 2) texture synthesizing and 3) style transfer via a random network instead of a well trained network as often done in the literature. The result shows that a random network can be used to perform these 3 tasks well, and hints that random networks can be used to perform comparative analysis between different net architectures.
OVERALL: This is an interesting paper and I enjoy looking at the resultant images. The paper does raise an interesting idea that one could use random nets for inverting representation in order to understand architectures; but, did not actually show any result to back up that idea. The texture synthesis and style transfer experiments with a random net demonstrate a new and more efficient way (without having to train a net) to perform these tasks.  However, the paper merely shows the results, but does not offer thorough investigation and insights into using random nets. It'd be great to have some of these questions answered in the paper:   1. is it surprising that inverting a random function/network works here?  2. how easy it is to invert a random nets compared to a trained net? 3. what are the disadvantages of using a random vs trained net? 4. how would the results change if performing these tasks with other architectures, e.g. (shallow) AlexNet, and (very deep) ResNet?  TECHNICAL QUALITY 1. Inverting deep representations. - For a fairer comparison (qualitatively and quantitatively) with Mahendran et al, one should use the same setup (i.e. AlexNet and/or regularizer). Why was VGG used here? - Inverting from layer1-5 has been shown to be easy (Mahendran et al 2014, Dosovitskiy & Brox 2016). I wonder if random weights would still work similarly for higher layers?  - This section would be much more impactful if the authors indeed inverted and compared different net architectures.  - One way to improve the inversion quality further is to optimize an image to match both the code and the texture (Gram matrix) of a target image.  2. Textures synthesis: the results looks qualitatively great, and interesting as it's in contrast with the finding by Gatsy et al. - I wonder if it would produce a large variation (i.e. does starting from different random images produce very different samples)  3. Style transfer: Fig. 5 shows a recognizable artifact (rough, dark blobs of colors) in ranVGG compared to [13] which is smoother / qualitatively better. It'd be great to show more investigation into why it happened and see if the trained weights matter a lot here or it's just due to different hyperparam settings.  NOVELTY Inverting and/or harnessing deep representations extracted from random networks has been previously studied in these two papers: - Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. arXiv preprint arXiv:1506.02753, 2015. - Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. arXiv preprint arXiv:1602.02644, 2016.  Their approach is not the same as the one in this paper. But the general idea is the same, thus, their works in using random nets should be mentioned. Using random nets in texture synthesis and style transfer is novel/original, afaik. There is this concurrent texture synthesis work with similar ideas though: https://arxiv.org/pdf/1505.07376v3.pdf.  POTENTIAL IMPACT The texture synthesis result shows high practical impact to the community. However, the representation inversion result is not very surprising and the authors have not demonstrated how such a random net can be used to understand different net architectures (as stated in the Intro & Discussion).  CLARITY & PRESENTATION Overall, the paper is nicely written and easy to read. I see a few places that could be improved for clarity and presentation:  L61-63: "Could we do deep visualization using untrained nets": the authors did not describe the scope of "deep visualization", but if it includes synthesizing a preferred images for neurons (see Nguyen et al 2016), then a random net wouldn't work. L16-17: The authors state that this proposed method sheds light on new understandings of deep vis, but the paper did not really discuss this in depth. L56-57: The purpose of a visualization technique is to enhance some understanding, thus I'm not sure what it's meant here.