Self-supervised Learning of Motion Capture
Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.
This work addresses the problem of motion capture for monocular videos in the wild, where we are interested in estimating human poses in RGB videos. Instead of directly optimizing the parameters of a 3D human model to reduce the error measured by the metrics such as segmentation, optical flow, or the coordinates of key points, this paper proposes an end-to-end trainable framework to predict the parameters of the SMPL model. The optimization objective includes the projection error of key points, optical flow, and foreground-background segmentation. The model is tested on Surreal dataset and H3.6M dataset, compared with a pre-trained model and a direct optimization framework.
[Strengths]
- The paper proposes a set of promising objectives which can be utilized to fine-tune the models on real-world datasets, which are pre-trained on synthetic datasets.
- Figures are well-drawn and do help readers understand the proposed framework.
[Weakness]
- This paper is poorly written. 
   - The motivation, the contribution, and the intuition of this work can barely be understood based on the introduction.
   - Sharing the style of citations and bullet items is confusing.
   - Representing a translation vector with the notation $t^{i} = [x^{i}, y^{i},  z^{i} ] ^{T}$ is usually more preferred.
- The experimental results are not convincing. 
   - The descriptions of baseline models are unclear.
   - Comparing the performance of the methods which directly optimize the mesh parameters, rotation, translation, and focal length according to the metric provided (projection error) doesn't make sense since the objective is in a different domain of the measurement of the performance.
   - Comparing the performance of the model only pre-trained on synthetic data is unfair; instead, demonstrating that the proposed three projection errors are important is more preferred. In other words, providing the performance of the models pre-trained on synthetic data but fine-tuned on real-world datasets with different losses is necessary. 
- The reason of claiming that it is a supervised learning framework is unclear. In my opinion, the supervision signals are still labeled.
[Reproducibility]
The proposed framework is very simple and well explained with sufficient description of network parameters and optimization details. I believe it's trivial to reproduce the results.
[Overall]
In term of the proposed framework, this paper only shows the improvement gained of fine-tuning the model based on the proposed losses defined by the reprojection errors of key points, optical flow, and foreground-background segmentation.
Taking into account that this work does show that fine-tuning the model pre-trained on synthetic datasets on real-world video clips improves the performance especially, it's still a convicting article. 
In sum, as far as I am concerned this work makes a contribution but is insufficient.