leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations
in appearance, scene and pose. We adopt a Multiple Instance Learning (MIL)arXiv:2007.10703v1  [cs.CV]  21 Jul 2020
2 A. Arnab et al.
Training dataPrediction
Labels: 
sit, talk to (people), watch (a person), carry/hold (an object), listen to (a person)
 sit, carry/hold (an object), listen to (a person), talk to (people), watch (a person)
sit, talk to (people), watch (a person)
sit, listen to (a person), watch (a person)
time
Fig. 1. We propose a method to train a spatio-temporal action detector using only
weak, video-level labels on challenging, real-world datasets. Note that the video-level
labels that we have may apply to multiple people in the video, and that these labels
may only be active for an unannotated time interval of the input clip.
framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid