Associative Embedding: End-to-End Learning for Joint Detection and Grouping
We introduce associative embedding, a novel method for supervising convolutional neural networks for the task of detection and grouping. A number of computer vision problems can be framed in this manner including multi-person pose estimation, instance segmentation, and multi-object tracking. Usually the grouping of detections is achieved with multi-stage pipelines, instead we propose an approach that teaches a network to simultaneously output detections and group assignments. This technique can be easily integrated into any state-of-the-art network architecture that produces pixel-wise predictions. We show how to apply this method to multi-person pose estimation and report state-of-the-art performance on the MPII and MS-COCO datasets.
The paper proposes a method for the task of multi-person pose estimation. It is based on the stacked hour glasses architecture for single person pose estimation which is combined with an associative embedding. Body joint heatmaps are associated with tag heatmaps indicating which individual the joint belongs to. A new grouping loss is defined to assess how well predicted tags agree with ground truth joint associations. The approach is evaluated on the MPII multi-person and on MS-COCO, achieving state of the art performance for both of them.
Strengths:
- State-of-the-art performance is achieved for the multi-person pose estimation problem on MPII Multi-person and MS-COCO.
Weakness:
- The paper is rather incremental with respect to [31]. The authors adapt the existing architecture for the multi-person case producing identity/tag heatmaps with the joint heatmaps.
- Some explanations are unclear and rather vague. Especially, the solution for the multi-scale case (end of Section 3) and the pose refinement used in section 4.4 / table 4. This is important as most of the improvement with respect to state of the art methods seems to come from these 2 elements of the pipeline as indicated in Table 4.
Comments:
The state-of-the-art performance in multi-person pose estimation is a strong point. However, I find that there is too little novelty in the paper with respect to the stacked hour glasses paper and that explanations are not always clear. What seems to be the key elements to outperform other competing methods, namely the scale-invariance aspect and the pose refinement stage, are not well explained.