12 A. Arnab et al.
Fig. 4. Qualitative examples on UCF101-24. Note that the bounding boxes are coloured
according to the identity of the track. The action label, and tube score are labelled
from the top-left of the bounding box. Further discussion is included in the text.
the online tubelet linking algorithm (we used the same method as [ 19]) has made
the identities of the two cyclists change after they occluded each other.
4.3 Experiments on AVA
In this section, we report what to our knowledge are the rst weakly-supervised
action detection experiments on AVA [ 14]. The AVA dataset labels keyframes
in a 15-minute video clip, where each keyframe is sampled every second ( i.e.at
1 Hz). The evaluation protocol of the AVA dataset measures the ability of an
action detection model to classify the actions occuring in a keyframe given the
temporal context around it.
We control the diculty of the weakly-supervised action recognition problem by combining the annotations from Nconsecutive keyframes into a single,
clip-level annotation. This eectively means that we are obtaining clip-level
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid