Published as a conference paper at ICLR 2018 GENERATIVE NETWORKS AS INVERSE PROBLEMS WITH SCATTERING TRANSFORMS
Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be optimized with a discriminator or an encoder. The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images. This embedding is computed with a wavelet Scattering transform. Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.
The paper proposes a generative model for images that does no require to learn a discriminator (as in GAN’s) or learned embedding. The proposed generator is obtained by learning an inverse operator for a scattering transform.
The paper is well written and clear. The main contribution of the work is to show that one can design an embedding with some desirable properties and recover, to a good degree, most of the interesting aspects of generative models. However, the model doesn’t seem to be able to produce high quality samples. In my view, having a learned pseudo-inverse for scattering coefficients is interesting on its own right. The authors should show more clearly the generalization capabilities to test samples. Is the network able to invert images that follow the train distribution but are not in the training set?
As the authors point out, the representation is non-invertible. It seems that using an L2 loss in pixel space for training the generator would necessarily lead to blurred reconstructions (and samples) (as it produces a point estimate). Unless the generator overfits the training data, but then it would not generalize. The reason being that many images would lie in the level set for a given feature vector, and the generator cannot deterministically disambiguate which one to match. 
The sampling method described in Section 3.2 does not suffer from this problem, although as the authors point out, a good initialization is required. Would it make sense to combine the two? Use the generator network to produce a good initial condition and then refine it with the iterative procedure.
This property is exploited in the conditional generation setting in:
Bruna, J. et al "Super-resolution with deep convolutional sufficient statistics." arXiv preprint arXiv:1511.05666 (2015).
The samples produced by the model are of poorer quality than those obtained with GAN’s. Clearly the model is assigning mass to regions of the space where there are not valid images.  (similar effect that suffer models train with MLE). Could you please comment on this point?
The title is a bit misleading in my view. “Analyzing GANs” suggests analyzing the model in general, this is, its architecture and training method (e.g. loss functions etc). However the analysis concentrates in the structure of the generator and the particular case of inverting scattering coefficients.
However, I do find very interesting the analysis provided in Section 3.2. The idea of using meaningful intermediate (and stable) targets for the first two layers seems like a very good idea. Are there any practical differences in terms of quality of the results? This might show in more complex datasets.
Could you please provide details on what is the dimensionality of the scattering representation at different scales? Say, how many coefficients are in S_5?
In Figure 3, it would be good to show some interpolation results for test images as well, to have a visual reference.
The authors mention that considering the network as a memory storage would allow to better recover known faces from unknown faces. It seems that it would be known from unknown images. Meaning, it is not clear why this method would generalize to novel image from the same individuals. Also, the memory would be quite rigid, as adding a new image would require adapting the generator.
Other minor points:
Last paragraph of page 1, “Th inverse \Phi…” is missing the ‘e’.
Some references (to figures or citations) seem to be missing, e.g. at the end of page 4, at the beginning of page 5, before equation (6). 
Also, some citations should be corrected, for instance, at the end of the first paragraph of Section 3.1: 
“… wavelet filters Malat (2016).” 
Sould be:
“... wavelet filters (Malat, 2016).” 
First paragraph of Section 3.3. The word generator is repeated.