tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set
r= 1. Finally, introducing our uncertainty-based loss function improves results
even further, obtaining a Video mAP of 35.0 at a threshold of 0.5. This is 80%
of the performance achieved by our fully-supervised baseline.
Person detections on UCF101-24 Note that for our weakly-supervised experiments, the person tubelets for training are obtained from a Faster-RCNN [ 37]
person detector that has only been trained on Microsoft COCO [ 26]. There is a
10 A. Arnab et al.
signicant domain gap between COCO and UCF, and the annotation protocol of
person boxes on UCF is also not consistent (for example, the bounding box for a
person riding a horse often includes the horse in UCF) with that of COCO. These
discrepancies are re
ected by the fact that our person detections used during
training only have a recall of 46.9% compared to the ground truth person boxes,
when using an IoU threshold of 0.5 to signify a correct match. Furthermore,
the precision of our person tubelets on the training set is only 21.1%. A major
contributing factor to this is that UCF action annotations are not exhaustive
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid