Uncertainty-Aware Weakly Supervised Action
Detection from Untrimmed Videos
Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia Schmid
Google Research
faarnab, chensun, anagrani, cordelias g@google.com
Abstract. Despite the recent advances in video classication, progress
in spatio-temporal action recognition has lagged behind. A major contributing factor has been the prohibitive cost of annotating videos frameby-frame. In this paper, we present a spatio-temporal action recognition
model that is trained with only video-level labels, which are signicantly
easier to annotate. Our method leverages per-frame person detectors
which have been trained on large image datasets within a Multiple Instance Learning framework. We show how we can apply our method in
cases where the standard Multiple Instance Learning assumption, that
each bag contains at least one instance with the specied label, is invalid using a novel probabilistic variant of MIL where we estimate the
uncertainty of each prediction. Furthermore, we report the rst weaklysupervised results on the AVA dataset and state-of-the-art results among
weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid