Non-parametric Structured Output Networks
Deep neural networks (DNNs) and probabilistic graphical models (PGMs) are the two main tools for statistical modeling. While DNNs provide the ability to model rich and complex relationships between input and output variables, PGMs provide the ability to encode dependencies among the output variables themselves. End-to-end training methods for models with structured graphical dependencies on top of neural predictions have recently emerged as a principled way of combining these two paradigms. While these models have proven to be powerful in discriminative settings with discrete outputs, extensions to structured continuous spaces, as well as performing efficient inference in these spaces, are lacking. We propose non-parametric structured output networks (NSON), a modular approach that cleanly separates a non-parametric, structured posterior representation from a discriminative inference scheme but allows joint end-to-end training of both components. Our experiments evaluate the ability of NSONs to capture structured posterior densities (modeling) and to compute complex statistics of those densities (inference). We compare our model to output spaces of varying expressiveness and popular variational and sampling-based inference algorithms.
The paper proposes Non-parametric Neural Networks (N3) a method that combines advantages of deep models for learning strong relations between input and output variables with the capabilities of probabilistic graphical models at modeling relationships between the output variables.
Towards this goal, the proposed method is designed based on three components:
a) a deep neural network (DNN) which learns the parameters of local non-parametric distributions conditioned on the input variables, b) a non-parametric graphical model (NGM) which defines a graph structure on the local distributions considered by the DNN. and, c) a recurrent inference network (RIN), which is trained/used to perform inference on the structured space defined by the DNN+NGM.
The proposed method is sound, well motivated and each of its components are properly presented. 
The method is evaluated covering a good set of baselines and an ablation study showing variants of the proposed method. The evaluation shows that state of the art results are achieved by the proposed method. However, due to its technical nature, there is a large number of variables and terms that complicate the reading at times. Still this is just a minor point.
Overall all the paper proposes a method with good potential in a variety of fields. However, one of my main concern is that its evaluation is conducted in the self-defined task of analyzing the statistics of natural pixels from the MS COCO dataset. In this regard, I would recommend adding more details regarding the evaluation, especially on how performance is measured for that task. Moreover, given that the evaluation is related with visual data, I would highly recommend evaluating their method on a more standard task from computer vision.
In addition, very recently a paper with the very same title as this manuscript ("non-parametric neural networks") came out, i.e. Phillip and Carbonell, ICLR'17, while the method and goal approached in this paper is different from that from ICLR, I would strongly suggest either, a) make an explicit and strong positioning of your paper w.r.t. Phillip and Carbonell, ICLR'17, or b) consider an alternative (or extended) title for your paper. In this regard, I consider that your paper is better suited for the N3 title, however, this is a conflict that should be addressed adequately.
Finally, given the potential of the proposed method I would encourage the authors to release the code related to the experiments presented in the manuscript. Are there any plans for releasing the code?
I believe, this should encourage the adoption of the proposed method by the research community. 
I would appreciate if the authors address the last three paragraphs in their rebuttal.