Invertibility of Convolutional Generative Networks from Partial Measurements
The problem of inverting generative neural networks (i.e., to recover the input latent code given partial network output), motivated by image inpainting, has recently been studied by a prior work that focused on fully-connected networks. In this work, we present new theoretical results on convolutional networks, which are more widely used in practice. The network inversion problem is highly non-convex, and hence is typically computationally intractable and without optimality guarantees. However, we rigorously prove that, for a 2-layer convolutional generative network with ReLU and Gaussian-distributed random weights, the input latent code can be deduced from the network output efficiently using simple gradient descent. This new theoretical finding implies that the mapping from the low-dimensional latent space to the high-dimensional image space is one-to-one, under our assumptions. In addition, the same conclusion holds even when the network output is only partially observed (i.e., with missing pixels). We further demonstrate, empirically, that the same conclusion extends to networks with multiple layers, other activation functions (leaky ReLU, sigmoid and tanh), and weights trained on real datasets.
This paper presents a new theoretical result on the invertibility of convolutional neural networks. The presented result is demonstrated (thoroughly in supplementary material) for a two-layer network, and claims are made about the empirical validity of the result on deeper networks. Invertibility is here intended as the recovery of latent code given partially observed network output (i.e. missing pixels), in the context of convolutional generative network. I think the direction of this paper's study is very interesting to better understand neural networks.  The theoretical result presented shows that the mapping from the latent code space to the network output (i.e. the image space) is bijective. This result is demonstrated only for a two-layer convolutional generative network with ReLU, while the general case is let as future work. Could you clarify the choice of a two-layer network? Is a two-layer network more tractable/significantly easier to analyze with respect to, for example, a three-layer network? What is the main difficulty in proving a more general result on deeper networks? Line 182 of the paper says: "This proof can be extended to deeper networks with the same machinery."  Could you clarify why you can only provide empirical evidence for the deeper network case? Why was the general result difficult to include in the paper? And if there is a specific difficulty, how would you suggest to address it? The same reasoning applies to the different activation functions: what was the main challenge in proving the results for different activation functions, and how would you address it? I think it would be important to clarify these points in the paper.   The proof of the main result mostly follows arguments in the archival paper “Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk” by Hand and Voroninski (cited), with some extensions to deal with the permutation applied to the weight matrices. While it seems that not much literature tackles this invertibility problem from the same angle, the cited “Towards Understanding the Invertibility of Convolutional Neural Networks” by Gilbert et al. (2017) is a relevant study, as well as other compressive sensing methods (e.g. the two cited works [4,5]).  How does the proposed approach compare with those works? A quantitative evaluation of the proposed solution, and comparative experiments with the other relevant works are missing. I would like to see a quantitative evaluation of the proposed solution on the experiments presented. I would also like to see how the proposed solution compares with other relevant approaches.  The paper is generally well written and pretty readable. Some claims seem generic, e.g. “we exploit the hierarchical nature of images and other natural signals by leveraging the powerful representation capability of deep learning.” (lines 63-65).  Could you clarify how this is supported by the rest of the paper? For example, only a two-layer network is considered for the proof of the main result, and no guarantees are provided for the validity of the result for deeper networks apart from empirical results (which are, in my opinion, not sufficient to support this claim).   Could you clarify how the experiments presented allow to see that the latent code was successfully recovered? I can only see reconstruction results, but there is no direct comparison between the “actual” latent code and the recovered one (which seems to be the point of this study).  One of the assumptions of the proposed approach is the Gaussian weight assumption. How restrictive is this assumption in practice? Could you provide examples where this assumption would not hold?   It is claimed that the proposed result "provides a sufficient condition to avoid mode collapses" (line 195). However this claim is not supported by experimental results. I would like to see, for example, an experiment where the mode collapse happens if the proposed solution is not applied.  Could you clarify how the permutation applied to the weight matrices affects the geometry of the latent space? How does this operation effectively act on the latent code representation? Can the permutation operation be interpreted as a way to disentangle features in the latent code? Are there information losses or computational costs associated to it?  After reading the author rebuttal, I tend to vote for the acceptance of this paper.