Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative
feature representations can be learned with unlabelled data. Examples in video
include cross-modal self-supervised learning by learning correspondences between
the audio and image streams readily available in videos [ 1,33,51], transcribed
speech [ 44] or using meta-data such as hashtags [ 10] as a form of weak labelling.
Self-supervised approaches, however, are complementary to our approach, as they
still require a limited amount of fully-labelled data for the nal task of interest.
4 A. Arnab et al.
In our weakly-supervised action detection scenario, we never have access to full,
spatio-temporal ground-truth annotations for a single training example.
3 Proposed Approach
As shown in Fig. 1, given a set of video clips, with only clip-level annotations of
the actions taking place, our goal is to learn a model to recognise and localise
these actions in space and time. Our method is based on Multiple Instance
Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid