The Return of the Gating Network: Combining Generative Models and Discriminative Training in Natural Image Priors
In recent years, approaches based on machine learning have achieved state-of-theart performance on image restoration problems. Successful approaches include both generative models of natural images as well as discriminative training of deep neural networks. Discriminative training of feed forward architectures allows explicit control over the computational cost of performing restoration and therefore often leads to better performance at the same cost at run time. In contrast, generative models have the advantage that they can be trained once and then adapted to any image restoration task by a simple use of Bayes' rule. In this paper we show how to combine the strengths of both approaches by training a discriminative, feed-forward architecture to predict the state of latent variables in a generative model of natural images. We apply this idea to the very successful Gaussian Mixture Model (GMM) of natural images. We show that it is possible to achieve comparable performance as the original GMM but with two orders of magnitude improvement in run time while maintaining the advantage of generative models.
The paper proposes a fast approximate inference approach for MAP estimation with the generative EPLL method [21]. To that end, discriminative training is used to replace the computationally expensive update step of the latent variables during alternating optimization. The experimental results show that while being 1-2 orders of magnitude faster, this approximate inference procedure causes only a relatively modest drop in performance as compared to previously-used ("exact") EPLL inference. # Positive I very much agree with the agenda of the paper to retain the benefits of generative approaches, as discussed at the beginning of the paper. As far as I see, the paper unfortunately doesn't really propose a generic method, since it only addresses a very specific problem of inference with the EPLL method. However, it is an interesting approach to use discriminative methods to approximate expensive steps during generative inference. Have you thought about how this can be used more generally? This would really interest me.It is interesting that EPLL yields similar performance with the proposed approximative inference procedure compared to the "exact" one. This has value, since EPLL yields state-of-the-art results and can now also be applied to large images with the proposed inference method (Figure 7 is quite impressive); I think the paper should emphasize this more.The puzzling part to me is why EPLL is seemingly the only generative approach that is competitive with discriminative methods; this would be interesting to understand. # Negative I think that the paper over-promises its contributions. I was really excited in the beginning while reading the introduction, but was then somewhat disappointed at the end. This is because the paper is somewhat misleading due to introduction being very generic and promising a general method to combine generative and discriminative approaches. However, the paper doesn't propose a "deep" integration between generative and discriminative methods. It requires standard generative training of the image prior, then trains a discriminative predictor to approximate expensive steps of generative inference.Furthermore, the paper then addresses only a very specific problem for performing MAP estimation with the EPLL method, which is as far as I see not easily applicable beyond this. Moreover, other generative approaches with filter-based image priors don't have the problem that updating the latent variables is computationally expensive [e.g., Krishnan and Fergus, "Fast image deconvolution using hyper-Laplacian priors", NIPS 2009].Although not really important for the actual contribution of the paper, the discussion of discriminative and generative approaches starting in l. 111 is not very nuanced. There are other things to consider than efficiency of test-time inference and "modularity", such as generative approaches being able to use different loss functions at test time, or performing marginalization as a principled approach to handle unknown random variables. # Clarity - The explanation of the gating network in sec. 3 is hard to understand, I think it needs some work to make it more clear. - Table 1 doesn't say for which test data these denoising results have been obtained. Are these different test sets in each paper? - Typo in Eq. 7: the weight w_i^j should be w_i^k in the denominator. - Typo in l. 102: "Hel-Or used..." -> "Hel-Or and Shaked used..." - Figure 2 is not clear since the different parts of the figure are not labeled. Also note that there seems to be a missing connection in the gating network on the right side: the bottom left node should be connected to the middle right node. - I find the title "return of gating network" not really appropriate. Also, "combining generative models and discriminative training in natural image priors" indicates to me that a generic method is proposed in the paper, which is not the case. # Miscellaneous - Note that "modularity" can also be achieved with discriminative approaches by training with a likelihood component that can later be replaced, which has been done by [Chen et al., "Revisiting loss-specific training of filter-based MRFs for image restoration", GCPR 2013, doi:10.1007/978-3-642-40602-7_30]. They trained a CRF via loss-based training for image denoising and then used it without re-training for deconvolution and super-resolution by swapping the likelihood term. - Note that EPLL inference doesn't actually use half-quadratic splitting as introduced by Geman and colleagues (for filter-based MRFs), cf. [Geman and Reynolds, "Constrained restoration and the recovery of discontinuities", PAMI 1992; Geman and Yang, "Nonlinear image recovery with half-quadratic regularization", TIP 1995]. I think the used method is best characterized as a quadratic penalty method for approximate constrained optimization [cf. Nocedal and Wright, "Numerical Optimization", Sec. 17.1].