CRF-CNN: Modeling Structured Information in Human Pose Estimation
Deep convolutional neural networks (CNN) have achieved great success. On the other hand, modeling structural information has been proved critical in many vision problems. It is of great interest to integrate them effectively. In a classical neural network, there is no message passing between neurons in the same layer. In this paper, we propose a CRF-CNN framework which can simultaneously model structural information in both output and hidden feature layers in a probabilistic way, and it is applied to human pose estimation. A message passing scheme is proposed, so that in various layers each body joint receives messages from all the others in an efficient way. Such message passing can be implemented with convolution between features maps in the same layer, and it is also integrated with feedforward propagation in neural networks. Finally, a neural network implementation of endto-end learning CRF-CNN is provided. Its effectiveness is demonstrated through experiments on two benchmark datasets.
This paper explores a CRF-CNN architecture for human pose estimation. The paper presents a an exposition of how to construct a CRF-CNN model and pass messages. Strong experimental results are provided.
The general idea of this work is clearly in a direction of interest to the community and the results look strong. However, there are a few aspects to this work that I find quite unclear.  If they were clearer this work would have much more potential for impact.  In particular, it is not clear enough if this work uses a truly 'end-to-end' approach (as stated for one of the contributions).  For example on line 46 it is stated that: "We show step by step how approximations are made to use an end-to-end learning CNN for implementing such CRF model."  On line 114 it is stated that "Therefore, the mapping from f to h ̃ can be implemented with one-layer transformation in a neural network and sigmoid is the activation function.  h ̃ is a new feature vector derived from f and f can be obtained from lower layers in a network."  However, there is no discussion about the procedure or the possibility of back propagating information into the CNN. It is stated on line 151 that "We use the fully convolutional implementation of VGG and the output of fc6 in VGG is used as the feature map".  It is therefore unclear precisely what is being done with respect to this issue.  * Is the CNN component being trained here through backpropagation, or is this a CRF on top of CNN features?  There has been a fair amount of work that has looked at these types of random field models for human pose prior to the explosion of interest in CNNs. See the CRF work of:  Kiefel, M. and Gehler, P.V., 2014, September. Human pose estimation with fields of parts. In European Conference on Computer Vision (pp. 331-346). Springer International Publishing.  and the references therein for pointers to some of the other prior work. I agree that while there are a number of examples of prior work that are very close to this formulation, there is a need for work that combines CNNs and CRFs in clear and clean ways.  The work presented here looks a lot like it boils down to taking features of a CNN as input into fairly well known CRF models, and therefore why not just use established inference methods like sum/max product for trees and mean field for loopy graphs  ?  It is also not clearly and explicitly stated why an inference procedure consisting of 'standard' sum/max product for a trees or mean-field for loopy graphs is or is not being used or compared with here. The motivation and justification of the proposed messages is unclear. These messages should be much more explicitly compared and contrasted with the corresponding established message passing equations. Ideally this comparison would be made both conceptually and empirically.  The abstract asserts: "A message passing scheme is proposed, so that in various layers each body joint receives messages from all the others in an efficient way".  While the manuscript is fairly clear and detailed regarding the CRF inference equations, these equations are not clearly motivated and related to the well known inference procedures of loopy BP and mean field.  On line 109 it is stated that "p(h|I , Θ) in (11) is approximated by a product of independent Q(hi |I , Θ) in (13) and (14), similar to mean-field approximation."  Why not use mean field exactly? In the case of loopy graphs, does the proposed algorithm become loopy BP? Since some of the motivation presented here is related to approximating h~ with an expectation (ex. (15)), is there a connection between the proposed procedure and expectation propagation?  The results regarding 'flooding' vs serial updates in Table 3 seem like they should be somewhat unsurprising if viewed through the lens of formally derived mean field updates. i.e. According to mathematics of mean field, updates should be done serially - each local Q is updated based on the current Q of neighbours. If viewed through the lens of loopy BP, loopy BP has been well studied and a more detailed placement of the procedure here with known results from loopy inference should be provided.  I think the paper would benefit greatly from comparing and contrasting both theoretically and empirically the proposed scheme with established mean field and loopy BP update methods.   The general idea of merging CRFs and CNNs is certainly of current interest and the quality of the exposition and exploration here will have an important influence on the long term potential for this paper to have impact.  The author response on these issues could alter my numerical ratings above.