MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.
Authors propose a method to generate training samples for the task of 3D pose estimation from single monocular images. To this end they utilize 2 data sources: 3D motion capture data and images annotated with 2D poses. For a given 3D pose and a camera viewpoint they retrieve N images (where N is number of joints), which have the most similar local geometry around respective joints as the 2D projection of the source 3D pose. Then, they stitch the N images into a coherent one with pose aware smoothing. The resulting synthetic dataset is added to the existing dataset which is annotated with 3D poses. 3D poses are clustered into K=5000 pose classes and a holistic CNN is trained for a classification task. At test time 3D pose is estimated as average pose of top scoring classes. This method achieves state-of-the-art performance on Human3.6M dataset. 
- Technical quality.  Authors compare their method to a very recent state-of-the-art method and use the same evaluation protocol. They present detailed evaluation of how synthetically generated data fairs against real data and how using them both together improves performance. Additionally authors demonstrate the advantage of formulating 3D pose estimation as a classification problem over direct regression to joint coordinates. This is facilitated by a large number of training samples, which they show is crucial for the classification CNN: in the small data regime full body regression actually outperforms 3D pose classification.  However, authors only evaluate their method using evaluation protocol described in [17, 40]. It would be nice to show how method compares to [18, 19] which use a different protocol on Human3.6M, even though it is an easier problem according to [17].  - Novelty/Originality  To the best of my knowledge this work is the first to synthesize training samples from real world images for the task of human pose estimation. This is an alternative approach to rendering images given the 3D pose of a subject. Despite the fact that generated images do not match real-world images in realism, using them is beneficial.  - Potential impact or usefulness  In order to train Deep Neural Networks large datasets are required, and this work explores how to generate such training sets. Therefore it can potentially be interesting to the community.  Since the method allows to generate diverse samples for 2D pose estimation as well, showing results on 2D pose estimation would demonstrate whether the proposed approach is applicable to other problems. Thus an experiment with augmenting typical 2D pose training sets such as MPII that shows an improvement in 2D pose estimation would be of great value. Otherwise, quantitative evaluation on only one dataset slightly questions the generality of the approach.  The evaluation on LSP while interesting, does not give insight of how the proposed data augmentation method can push the 2D pose estimation, because it does not use an established evaluation metric.  - Clarity and presentation  The data generation method is described with great detail and I had no problem understanding it. The paper reads well overall, however there are a number of spelling mistakes to be corrected.  - Final assessment  Overall, I like the paper. Authors clearly demonstrate that adding training samples, generated with their method, improves performance of 3D pose estimation  using a classification-based CNN. However I feel that quantitative evaluation on just one benchmark is not enough to conclude on generality of the data generation method, especially because generated images do not appear entirely realistic.