Towards Metamerism via Foveated Style Transfer
The problem of visual metamerism is defined as finding a family of perceptually indistinguishable, yet physically different images. In this paper, we propose our NeuroFovea metamer model, a foveated generative model that is based on a mixture of peripheral representations and style transfer forward-pass algorithms. Our gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder which allows us to encode images in high dimensional space and interpolate between the content and texture information with adaptive instance normalization anywhere in the visual field. Our contributions include: 1) A framework for computing metamers that resembles a noisy communication system via a foveated feed-forward encoder-decoder network -We observe that metamerism arises as a byproduct of noisy perturbations that partially lie in the perceptual null space; 2) A perceptual optimization scheme as a solution to the hyperparametric nature of our metamer model that requires tuning of the image-texture tradeoff coefficients everywhere in the visual field which are a consequence of internal noise; 3) An ABX psychophysical evaluation of our metamers where we also find that the rate of growth of the receptive fields in our model match V1 for reference metamers and V2 between synthesized samples. Our model also renders metamers at roughly a second, presenting a ×1000 speed-up compared to the previous work, which allows for tractable data-driven metamer experiments.
Summary:
The paper proposes a fast method for generating visual metamers – physically different images that cannot be told apart from an original – via foveated, fast, arbitrary style transfer. The method achieves the same goal as an earlier approach (Freeman & Simoncelli 2011): locally texturizing images in pooling regions that increase with eccentricity, but is orders of magnitude faster. The authors perform a psychophysical evaluation to test how (in)discriminable their synthesized images are amongst each other and compared with originals. Their experiment replicates the result of Freeman & Simoncelli of a V2-like critical scaling in the synth vs. synth condition, but shows that V1-like or smaller scaling is necessary for the original vs. synth condition.
I reviewed an earlier version of this paper for a different venue, where I recommended rejection. The authors have since addressed some of my concerns, which is why I am more positive about the paper now.
Strengths:
+ The motivation for the work is clear and the implementation straightforward, combining existing tools from style transfer in a novel way.
+ It's fast. Rendering speed is indeed a bottleneck in existing methods, so a fast method is useful.
+ The perceptual quality of the rendered images is quantified by psychophysical testing.
+ The role of the scaling factor for the pooling regions is investigated and the key result of Freeman & Simoncelli (pooling regions scale with 0.5*eccentricity) is replicated with the new method. In addition, the result of Wallis et al. (2018) that lower scale factors are required for original vs. synth is replicated as well.
Weaknesses:
- Compared with earlier work, an additional fudge parameter (alpha) is introduced. It is not clear why it is necessary and it complicates interpretation.
- The paper contains a number of sections with obscure mathiness and figures that I can't follow and whose significance is unclear.
Conclusion:
The work is well motivated, the method holds up to its promise of being fast and is empirically validated. However, it feels quite ad-hoc and the writing of the paper is very obscure at various places, which leaves room for improvement.
Details:
- The motivation for introducing alpha not clear to me. Wasn't the idea of F&S that you can reduce the image to its summary statistics within a pooling region whose size scales with eccentricity? Why do you need to retain some content information in the first place? How do images with alpha=1 (i.e. keep only texture) look?
- Related to above, why does alpha need to change with eccentricity? Experiment 1 seems to suggest that changing alpha leads to similar SSIM differences between synths and originals as F&S does, but what's the evidence that SSIM is a useful/important metric here?
- Again related to above, why do you not use the same approach of blending pooling regions like F&S did instead of introducing alpha?
- I would like to know some details about the inference of the critical scaling. It seems surprisingly spot on 0.5 as in F&S for synth vs. synth, but looking at the data in Fig. 12 (rightmost panel), I find the value 0.5 highly surprising given that all the blue points lie more or less on a straight line and the point at a scaling factor of 0.5 is clearly above chance level. Similarly, the fit for original vs. synth does not seem to fit the data all that well and a substantially shallower slope seems equally plausible given the data. How reliable are these estimates, what are the confidence intervals, and was a lapse rate included in the fits (see Wichmann & Hill 2001)?
- I don't get the point of Figs. 4, 13 and 14. I think they could as well be removed without the paper losing anything. Similarly, I don't think sections 2.1 and the lengthy discussion (section 5) are useful at all. Moreover, section 3 seems bogus. I don't understand the arguments made here, especially because the obvious options (alpha=1 or overlapping pooling regions; see above) are not even mentioned.
- How is the model trained? Do the authors use the pre-trained model of Huang & Belongie or is the training different in the context of the proposed method? I could only find the statement that the decoder is trained to invert the encoder, but that doesn't seem to be what Huang & Belongie's model does and the paper does not say anything about how it's trained to invert. Please clarify.
- At various places the writing is somewhat sloppy (missing words, commas, broken sentences), which could have been avoided by carefully proof-reading the paper.