Proximal Deep Structured Models
Many problems in real-world applications involve predicting continuous-valued random variables that are statistically related. In this paper, we propose a powerful deep structured model that is able to learn complex non-linear functions which encode the dependencies between continuous output variables. We show that inference in our model using proximal methods can be efficiently solved as a feedfoward pass of a special type of deep recurrent neural network. We demonstrate the effectiveness of our approach in the tasks of image denoising, depth refinement and optical flow estimation.
The paper presents a way to train a certain class of energy-based models commonly used in computer vision tasks with potential functions based on deep learning outputs.  The proposed method is a backpropagation-through-inference (BPTI) approach, where an iterative algorithm for optimizing the energy is unrolled for a finite number of steps, and each step is treated as a regular computation layer in a deep neural net.  Although BPTI has been applied in many previous works, the present work is novel principally in that it applies to a general class of energy-based models over continuous-valued states, whereas previous works either applied only to discrete problems, or more restricted classes of continuous problems.  Experiments show state-of-the art results on several applications. 
The work addresses a problem of significant current interest, since it addresses the issue of how to combine state-of-the-art variational methods for computer vision tasks with deep learning methods, which have rapidly become the de-facto choice for nearly all applications.  The proposed solution is practical, elegant, computationally efficient, and applicable to a wide variety of problems.  The experimental results are also state-of-the-art for several tasks.  Therefore, I believe that this work could have a significant practical impact on the field.  My only, relatively minor, reservation with the work is its novelty.  If you are a specialist in state-of-the-art convex optimization methods for computer vision tasks, and you know about structured prediction models for deep learning, then you could argue that this is a fairly obvious thing to do, since it is basically backpropagation-through-inference (BPTI) applied to primal-dual methods for convex variational problems.  On the other hand, this would be novel to anyone who is not a specialist in both those fields.  So, one could argue that the novelty of this method is mainly of the type of cross-pollination between two fields that should have more interaction, but there are no fundamentally new methods here.  Delving a little deeper into this issue, one might consider what the potential alternatives to this method may be, and how obvious each of those alternatives may be.  Prior work has mostly approached similar problems from a graphical models perspective, in which applying BPTI to belief propagation is the obvious solution.  Relative to that, the main innovation of this work is in defining a BP-like inference algorithm that works for an important class of graphical models over continuous variables.  Although graphical models people tend to shy away from continuous variables, BP (in this case, the max-product variant) still works conceptually in the continuous case.  The main difficulty is that the messages become functions, which are generally hard to represent.  However, if one assumes that the local minimizations have stable closed-form solutions, then max-product BP may still be applicable.  It would be interesting to see whether this is feasible, and if so, what relationship this method would have to the proposed method.  Another potential baseline would be block coordinate descent: i.e., instead of using the primal-dual method, alternatively fix blocks of variables and optimize them jointly.  For example, given “4-connected” cliques with variables on a regular grid, one could perform a method analogous to red-black Gauss-Seidel iteration, alternatively fixing ‘red’ and ‘black’ pixels.  This would reduce to independent, pointwise optimizations over the grid.  If one again assumes that these pointwise optimizations have closed-form solutions, then BPTI should be applicable to this method as well.  Experimentally, the work seems to be strong, showing state-of-the-art results compared to other methods, although I am unfortunately not that familiar with these benchmarks and many of the other methods.  One small caveat is that the image denoising dataset seems to be fairly small, and the Gaussian noise assumption a bit too simple, although it has apparently been used as a benchmark for other methods as well.  It would also be nice to have an idea of the computational performance for the depth refinement and optical flow tasks, although I assume they would be similar to the denoising case.  A summary of my review along each of the evaluation metrics is provided below.  Technical quality: The method is sound, and the experimental results are thorough and state-of-the-art.  Novelty: Although no fundamentally new methods are developed here, this work constitutes a clever way to combine state-of-the-art techniques from different fields (high-dimensional convex optimization and deep learning).  Potential impact: Due to the relative simplicity of the method, its computational efficiency, and state-of-the-art results, I think the method could (or at least should) be adopted widely for practical applications.  Clarity: The paper is written in a straightforward way and is easy to read.  ============ REVISION POST-REBUTTAL:  After the review period, I happened to find some uncited work (see [A] below) that, at least superficially, seems similar.  That work also involves doing backprop through a proximal method. I also mentioned in my review the possibility of doing backprop through coordinate descent, and it seems that this idea was also tried in [A].   So, I am now a little more worried about the technical novelty of this method.  A future revision of the paper should definitely cite [A] and discuss the differences between this work and that.  [A] K. Gregor and Y. Lecun. Learning fast approximations of sparse coding. ICML 2010.