Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images
We present MubyNet -a feed-forward, multitask, bottom up system for the integrated localization, as well as 3d pose and shape estimation, of multiple people in monocular images. The challenge is the formal modeling of the problem that intrinsically requires discrete and continuous computation, e.g. grouping people vs. predicting 3d pose. The model identifies human body structures (joints and limbs) in images, groups them based on 2d and 3d information fused using learned scoring functions, and optimally aggregates such responses into partial or complete 3d human skeleton hypotheses under kinematic tree constraints, but without knowing in advance the number of people in the scene and their visibility relations. We design a multi-task deep neural network with differentiable stages where the person grouping problem is formulated as an integer program based on learned body part scores parameterized by both 2d and 3d information. This avoids suboptimality resulting from separate 2d and 3d reasoning, with grouping performed based on the combined representation. The final stage of 3d pose and shape prediction is based on a learned attention process where information from different human body parts is optimally integrated. State-of-the-art results are obtained in large scale datasets like Human3.6M and Panoptic, and qualitatively by reconstructing the 3d shape and pose of multiple people, under occlusion, in difficult monocular images.
This paper presents a multi-stage deep network for recovering 3D poses and shapes of multiple people from a single image. Unlike the top-down approaches that use human detectors to decompose the problem into single-person subtasks, this paper follows a recent bottom-up method that detects all the necessary local features so that the multi-person sensing problem can be solved jointly by graph matching. Given extracted features from the input image, the first stage predicts 2D keypoint heatmaps and 3D pose hypotheses on those limb pixels. The second stage first extracts local features from all the limb candidates and predicts their scores of being valid limbs or connections, and then solves a binary integer programming problem based on limb scores and the skeleton kinematic tree to group keypoints into 2D skeletons. In the third stage, the predicted 3D pose hypotheses on the predicted limb pixels are combined to synthesis the final 3D pose of each skeleton. The body shape is based on the SMPL model. The fitted shape and pose parameters (alpha and beta) are predicted from 3D skeletons via a differentiable mesh animation function. As the alpha and beta are unsupervised, GMM and L2 priors are used to regularize them. Results are first reported on a subset of Human3.6M with a single person in each image. Multi-person results are reported on CMU Panoptic datasets with comparisons to previous optimization-based methods.   +This paper presents a complex deep network system for single-image 3D human sensing. All the components are clearly explained and corresponding chosen techniques are also well motivated, so the overall pipeline seems pretty reasonable. All the math notations are also well defined.  +The limb scoring network is pretty interesting. Previous bottom-up approaches either predict dense limb vectors (PAFs) or some forms of per-pixel hough votes. The limb scoring network instead simplifies the problem into a light-weighted limb hypothesis verification problem based on sparsely sampled features.  +The unsupervised SMPL modeling fitting is also interesting.  +The good quantitative results on Human3.6M and CMU Panoptic datasets verify the effectiveness of the proposed method.  -As it is a system-level work, such individual component may lack sufficient evaluations. For example, the limb scoring network bears some similarities with the part affinity fields in OpenPose. Although this paper argues that PAFs have some artificial parameters, e.g. the width of each PAF and need greedy grouping, it will be great to show some examples to compare the grouping quality and also present quantitative evaluations on 2D pose detection. I think it will strengthen the contribution and shed light on how better 2D pose detection can improve the overall results. The in-network SML fitting also needs some justifications. Experimental comparisons are need to demonstrate the claimed advantages over direct alpha-beta regression in [14]. -Some technical details may be missing. How is the person’s scene translation t being predicted? It is quite an important parameter. If the scene translations of two adjacent people are too close, there will be mesh penetration. But I didn’t see such results in the paper. I’m curious how this paper solved it or just didn’t present such results. Related to this question, since it is multi-person 3d pose detection, so how the 3D pose of each person is represented? The joint locations relative to root or joint locations in the scene? What is the value of N_samples? It is not clear from Fig. 3(b) that how the samples are extracted. A better figure is needed.  -It will be great if this paper can present some failure cases which will inspire future works.