centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple
Instance Learning framework in conjunction with human detections. The authors,
however, assumed that sparse spatial supervision was present ( i.e.bounding
boxes for a small subset of frames in the action tube), unlike our method which
requires video-level labels alone.
We also note that many approaches have addressed temporal action detection
(localising actions in time but not space) with only video-level tags as supervision [ 48,42,31,34]. UntrimmedNets [ 48] uses a network with two branches, a
classication module to perform action classication and a selection module to
select relevant frames.Hide-and-Seek [ 42] obtains more precise temporal boundaries by forcing the network to attend to more discriminative frames by randomly
hiding parts of videos. However, these methods are trained and evaluated on
datasets such as ActivityNet [ 4] and THUMOS14 [ 18], which contain mostly one
action per video, and are thus signicantly less challenging than datasets such as
AVA [14] which we evaluate on.
Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid