spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.
CNN feature extractorROI AlignClassiﬁer
Uncertainty-aware MILLoss ℒMultiple Instance Learning Bag
Probabilities, pUncertainties,  σ
Bag-level label
…
stand, listen to (a person), watch (a person), 
walk, carry/hold (an object) …
Fig. 2. Overview of our approach for training an action detector in a weakly-supervised
manner using multiple instance learning: Each bag consists of all the tubelets that have
been extracted from the video-clip. These tubelets are obtained using an o-the-shelf
person detector which has not been trained on the dataset of interest. These tubelets
act as proposals for a Fast-RCNN style detector operating on a sequence of rgb images.
The predictions for each of the tubelets in the bag are then aggregated together, and
compared to the bag-level label. Uncertainty estimates produced by the network are
used to compensate for noise in the bag-level labels during training.
detections. Temporal localisation, on the other hand, is performed by linking
person tubelets through the video as commonly done in the literature [ 19,41,52,6],
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid