time given only video-level annotations. To facilitate this, we leverage a person
detector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].
Concretely, we run a person detector on our training videos, and create person
tubelets which are person detections over Kconsecutive frames in the video.
Our bag for multiple instance learning thus consists of all the tubelets within a
video, and is annotated with the video-level labels that we have as supervision,
as illustrated in Fig. 2. Note that the size of the bag varies for every video clip, as
the bag size is determined by the length of the video and the number of detected
people.
As shown in Fig. 2, our network architecture for this task is a Fast-RCNN [ 12]
style detector that has been extended temporally. Given a video clip of Kframes,
and proposals which in our case are person detections, the network classies the
action(s) taking place at the centre frame of each proposal, given the temporal
context of the K1 frames around it.
Note that the spatio-temporal localisation task is eectively factorised: the
spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid