Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
This paper proposes a new hybrid architecture that consists of a deep Convolutional Network and a Markov Random Field. We show how this architecture is successfully applied to the challenging problem of articulated human pose estimation in monocular images. The architecture can exploit structural domain constraints such as geometric relationships between body joint locations. We show that joint training of these two model paradigms improves performance and allows us to significantly outperform existing state-of-the-art techniques.
TECHNICAL CONTENT & CLARITY:The "deep convolutional network" part-detectors of Sec. 3.1 seem like a sensible application of recently developed deep learning methods.  There does not appear to be significant novelty, but the architecture seems carefully engineered and is clearly described.  One comment, if the "approximate" architecture in Figure 4 is used for the actual results, why bother to show the more complex architecture in Figure 3?  There are no conceptual principles presented as to why one should be better than the other; the validation is experimental, so why not focus your explanatory effort on the thing you actually experiment with?In Sec. 3.2, there is no valid joint distribution defined on the body part locations, and no non-trivial graphical structure; thus it seems inappropriate to call this a "graphical model".  Instead there is a single round of message passing between all part pairs, as illustrated in Figure 5.  This message passing has a form motivated by the sum-product BP algorithm, but need not correspond to BP (for example in a valid graphical model there would be constraints relating f|s and s|f which are not enforced here), and is not iterated to any mutually-consistent fixed point.  A comment after Equation (2) says this formulation is not "exactly equivalent to an MRF", but I would argue more strongly, and say that the final formulation is rather distant from an MRF.  The claims in earlier parts of the paper on integration with graphical models should be moderated to reflect this.For related work which is more similar to this approach, see the "cascaded classification models" of Heitz 2008, or more distantly the "sum-product networks" developed by Domingos and colleagues.  The approach is rather different from work like Bergtholdt 2010, which carefully treats the issues of inference and learning in fully-connected part-based graphs.EXPERIMENTS:Results on two contemporary pose estimation datasets are very good, improving on several strong baselines.  A careful internal comparison also shows relative benefits of adding the final spatial message-passing layer, and doing a final round of back-propagation training to refine the full set of model parameters.  These results are certainly the greatest strength of the paper.I would have liked to also see an evaluation on the "Buffy the Vampire Slayer" dataset, which has been very widely studied in the pose estimation literature, and would this provide a wider set of baselines.REFERENCES:G. Heitz, S. Gould, A. Saxena, and D. Koller (2008). "Cascaded Classification Models: Combining Models for Holistic Scene Understanding." Advances in Neural Information Processing Systems (NIPS 2008).Bergtholdt, Kappes, Schmidt, Schnorr, A Study of Parts-Based Object Class Detection Using Complete Graphs.  International Journal of Computer Vision, Volume 87 Issue 1-2, March 2010, Pages 93 - 117.