weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video
understanding, mulitple instance learning
1 Introduction
Video classication has witnessed great advances recently due to large datasets
such as Kinetics [ 20] and Moments in Time [ 30] which have enabled training
of specialised neural network architectures for video [ 5,9]. However, progress in
other video understanding tasks, such as spatio-temporal action detection, has
lagged behind in comparison. There are fewer datasets for action recognition,
which are also signicantly smaller than their video-classication counterparts. A
reason for this is the exorbitant cost of annotating videos with spatio-temporal
labels { each frame of an action has to be manually labelled with a bounding
box. Moreover, annotating temporal boundaries of actions is not only arduous,
but often ambiguous with annotators failing to reach consensus about the start
and end times of an action [6,39].
In this paper, we propose a method to train spatio-temporal action detectors
using only weak, video-level annotations as shown in Fig. 1. To achieve this, we
leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid