the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.
Based on the results in the left subﬁgure of Figure 4, we can draw several conclusions. First, we
note that when there are very few labeled videos (i.e., 5), all three baselines perform poorly (leftmost
ﬁgure). This indicates that in this setting there is not enough data to learn an effective pose detection
model. Second, we observe that when the number of labeled videos is somewhat reasonable (e.g.,
50100), our approach signiﬁcantly outperforms the "GT (1x)" baseline, and is only slightly worse
relative to the "GT (7x)" baseline. As we increase the number of labeled videos, the gaps among the
three methods shrink, suggesting that the model becomes saturated.
7
Table 2: Multi-person pose estimation results on the validation and test sets of PoseTrack2017
and PoseTrack2018 datasets. Even though our model is designed to improve pose detection in
scenarios involving sparsely-labeled videos, here we show that our temporal pose aggregation scheme
during inference is also useful for models trained on densely labeled videos. We improve upon the
state-of-the-art single-frame baselines [23, 27, 56].
Modern approaches for multi-person pose estimation in video require large
amounts of dense annotations. However, labeling every frame in a video is
costly and labor intensive. To reduce the need for dense annotations, we
propose a PoseWarper network that leverages training videos with sparse
annotations (every k frames) to learn to perform dense temporal pose
propagation and estimation. Given a pair of video frames---a labeled Frame A
and an unlabeled Frame B---we train our model to predict human pose in Frame A
using the features from Frame B by means of deformable convolutions to
implicitly learn the pose warping between A and B. We demonstrate that we can
leverage our trained PoseWarper for several applications. First, at inference
time we can reverse the application direction of our network in order to
propagate pose information from manually annotated frames to unlabeled frames.
This makes it possible to generate pose annotations for the entire video given
only a few manually-labeled frames. Compared to modern label propagation
methods based on optical flow, our warping mechanism is much more compact (6M
vs 39M parameters), and also more accurate (88.7% mAP vs 83.8% mAP). We also
show that we can improve the accuracy of a pose estimator by training it on an
augmented dataset obtained by adding our propagated poses to the original
manual labels. Lastly, we can use our PoseWarper to aggregate temporal pose
information from neighboring frames during inference. This allows our system to
achieve state-of-the-art pose detection results on the PoseTrack2017 and
PoseTrack2018 datasets. Code has been made available at:
https://github.com/facebookresearch/PoseWarper.
Gedas Bertasius Christoph Feichtenhofer Du Tran Jianbo Shi Lorenzo Torresani