What does it take to generate natural textures?
Natural image generation is currently one of the most actively explored fields in Deep Learning. Many approaches, e.g. for state-of-the-art artistic style transfer or natural texture synthesis, rely on the statistics of hierarchical representations in supervisedly trained deep neural networks. It is, however, unclear what aspects of this feature representation are crucial for natural image generation: is it the depth, the pooling or the training of the features on natural images? We here address this question for the task of natural texture synthesis and show that none of the above aspects are indispensable. Instead, we demonstrate that natural textures of high perceptual quality can be generated from networks with only a single layer, no pooling and random filters.
The framework of Gatys et al. demonstrated that correlation statistics (empirical Gram matrices) of deep feature responses provide an excellent characterisation of visual textures. This paper investigates in detail which kind of deep or shallow networks may work well in this framework. One of the main findings is that that very shallow nets, consisting of a single filter bank with random weights, work surprisingly well, and for simple and regular textures may produce results which are visually superior to complex data-adapted filters such as the ones in networks like VGG-19. More broadly, the paper contains an interesting and informative discussion on the strength and limitations on such methods for texture synthesis. Figure 4 shows that the optimisation of images with respect to shallow filter banks may result in texture images that have a lower VGG-19 loss than optimising the VGG-19 objective directly. This is imputed to the difficulty of optimising the highly non-linear VGG-19 cost function, which is a reasonable explanation. In the new supplementary material, the authors show that better optimisation results can be obtained by initialising the VGG-19-based optimisation with the shallow network optimisation results, which is a useful complement to the original experiments. The main limitation of the paper is that it does not systematically compare different methods against a quantifiable objective. It is trivial to define image statistics that would allow to simply generate an exact copy of any reference texture, hence with very good visual quality. Such trivial statistics would also be very shallow. The aim is instead to capture a texture distribution, and measuring how well a method meets this challenge remains an open problem. Hence, while the empirical results seem to confirm the intuition that simple statistics are good enough for texture synthesis both in terms of quality and diversity (when compared to more complex statistics), it is difficult to conclusively confirm that this is the case. The authors indicate that diversity could be measured in terms of entropy. This is reasonable, but, as they acknowledge in their answers to questions, difficult to do in practice. Furthermore, this would still not account for the other aspect of the problem, namely visual quality. They also suggest to perform a psychophysical assessment, which may be the only practical way of addressing this problem, but deem that to be material for future work. Overall, since evaluation of image generation is such an hard problem, I think the paper still has sufficient strengths to warrant publication in ICLR. Still, some form of psychophysical assessment would be useful to confirm the intuitions that, at present, can only be obtained by inspecting the figure sin the paper and in the supplementary material.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
What does it take to generate natural textures?
Natural image generation is currently one of the most actively explored fields in Deep Learning. Many approaches, e.g. for state-of-the-art artistic style transfer or natural texture synthesis, rely on the statistics of hierarchical representations in supervisedly trained deep neural networks. It is, however, unclear what aspects of this feature representation are crucial for natural image generation: is it the depth, the pooling or the training of the features on natural images? We here address this question for the task of natural texture synthesis and show that none of the above aspects are indispensable. Instead, we demonstrate that natural textures of high perceptual quality can be generated from networks with only a single layer, no pooling and random filters.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The framework of Gatys et al. demonstrated that correlation statistics (empirical Gram matrices) of deep feature responses provide an excellent characterisation of visual textures. This paper investigates in detail which kind of deep or shallow networks may work well in this framework. One of the main findings is that that very shallow nets, consisting of a single filter bank with random weights, work surprisingly well, and for simple and regular textures may produce results which are visually superior to complex data-adapted filters such as the ones in networks like VGG-19. More broadly, the paper contains an interesting and informative discussion on the strength and limitations on such methods for texture synthesis. Figure 4 shows that the optimisation of images with respect to shallow filter banks may result in texture images that have a lower VGG-19 loss than optimising the VGG-19 objective directly. This is imputed to the difficulty of optimising the highly non-linear VGG-19 cost function, which is a reasonable explanation. In the new supplementary material, the authors show that better optimisation results can be obtained by initialising the VGG-19-based optimisation with the shallow network optimisation results, which is a useful complement to the original experiments. The main limitation of the paper is that it does not systematically compare different methods against a quantifiable objective. It is trivial to define image statistics that would allow to simply generate an exact copy of any reference texture, hence with very good visual quality. Such trivial statistics would also be very shallow. The aim is instead to capture a texture distribution, and measuring how well a method meets this challenge remains an open problem. Hence, while the empirical results seem to confirm the intuition that simple statistics are good enough for texture synthesis both in terms of quality and diversity (when compared to more complex statistics), it is difficult to conclusively confirm that this is the case. The authors indicate that diversity could be measured in terms of entropy. This is reasonable, but, as they acknowledge in their answers to questions, difficult to do in practice. Furthermore, this would still not account for the other aspect of the problem, namely visual quality. They also suggest to perform a psychophysical assessment, which may be the only practical way of addressing this problem, but deem that to be material for future work. Overall, since evaluation of image generation is such an hard problem, I think the paper still has sufficient strengths to warrant publication in ICLR. Still, some form of psychophysical assessment would be useful to confirm the intuitions that, at present, can only be obtained by inspecting the figure sin the paper and in the supplementary material.
assistant
The framework of Gatys et al. demonstrated that correlation statistics (empirical Gram matrices) of deep feature responses provide an excellent characterisation of visual textures. This paper investigates in detail which kind of deep or shallow networks may work well in this framework. One of the main findings is that that very shallow nets, consisting of a single filter bank with random weights, work surprisingly well, and for simple and regular textures may produce results which are visually superior to complex data-adapted filters such as the ones in networks like VGG-19. More broadly, the paper contains an interesting and informative discussion on the strength and limitations on such methods for texture synthesis. Figure 4 shows that the optimisation of images with respect to shallow filter banks may result in texture images that have a lower VGG-19 loss than optimising the VGG-19 objective directly. This is imputed to the difficulty of optimising the highly non-linear VGG-19 cost function, which is a reasonable explanation. In the new supplementary material, the authors show that better optimisation results can be obtained by initialising the VGG-19-based optimisation with the shallow network optimisation results, which is a useful complement to the original experiments. The main limitation of the paper is that it does not systematically compare different methods against a quantifiable objective. It is trivial to define image statistics that would allow to simply generate an exact copy of any reference texture, hence with very good visual quality. Such trivial statistics would also be very shallow. The aim is instead to capture a texture distribution, and measuring how well a method meets this challenge remains an open problem. Hence, while the empirical results seem to confirm the intuition that simple statistics are good enough for texture synthesis both in terms of quality and diversity (when compared to more complex statistics), it is difficult to conclusively confirm that this is the case. The authors indicate that diversity could be measured in terms of entropy. This is reasonable, but, as they acknowledge in their answers to questions, difficult to do in practice. Furthermore, this would still not account for the other aspect of the problem, namely visual quality. They also suggest to perform a psychophysical assessment, which may be the only practical way of addressing this problem, but deem that to be material for future work. Overall, since evaluation of image generation is such an hard problem, I think the paper still has sufficient strengths to warrant publication in ICLR. Still, some form of psychophysical assessment would be useful to confirm the intuitions that, at present, can only be obtained by inspecting the figure sin the paper and in the supplementary material.