2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.
Initial approaches in the area used 3D sliding window detectors in conjunction
with handcrafted, volumetric features [ 21,25]. Current state-of-the-art approaches
are temporal extensions of object detection architectures [ 19,41,52,35] such as
Faster-RCNN [ 37] and SSD [ 27]. These approaches predict bounding boxes around
the action in a frame, using as input either a single frame along with optical
Uncertainty-Aware Weakly Supervised Action Detection 3

ow to capture temporal information [ 41,38] or multiple frames at the input to
provide temporal context [ 19]. The predicted bounding boxes are then linked
over time using an online, greedy algorithm or dynamic programming to create
spatio-temporal tracks. Our work builds on these methods by also utilising a
detection architecture and spatio-temporal linking. However, these approaches
all require bounding box annotations at each frame in the video whilst we only
use video-level labels which are signicantly cheaper to acquire.
Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid