framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of
label noise: this is introduced from using o-the-shelf person detectors which have
not been trained on the video-dataset of interest, and also the fact that we have
to sample tubelets from large bags in long videos due to memory constraints. In
both of these scenarios, the standard Multiple Instance Learning assumption [ 7],
that each bag contains at least one instance with the bag-level label, may be
violated. We are not aware of previous work that has explicitly addressed this
problem, and we do so with a probabilistic variant of MIL where we estimate the
uncertainty of an instance-level prediction.
Using our approach, we obtain state-of-the-art results among weakly-supervised
methods on the UCF101-24 dataset. Furthermore, we report, to our knowledge,
the rst weakly-supervised results on the AVA dataset (the only large-scale
dataset for spatio-temporal action recognition), where we also show the accuracy
trade-os when annotating video-clips for time intervals of varying durations.
2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid