Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation
time and cost. Relevant to our approach is the work of [ 6]. Che ron et al. [6] also
use person detections, and infer their action labels using a formulation based on
discriminative clustering [ 2]. Although their approach allows them to incorporate
dierent types of supervision, it eectively learns a linear classier on top of
pretrained, deep features. Our method in contrast is learned fully end-to-end.
Mettes et al. [29] also employed Multiple Instance Learning (MIL), but used
action proposals [ 47] instead of the human detections used by our work and [ 6].
However, [ 29], rely on additional cheap \point" annotations (a single spatial point
annotated for a subset of the frames which constitute the action) which also
ensures that the standard MIL assumption is not violated. In follow-up work [ 28],
the authors removed the need for \point" annotations by incorporating biases
(i.e.the presence of objects in the video, a bias that actions typically occur in the
centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid