Sub-clip duration (seconds)
FS 1 5 10 30 60 900
Frame AP 24.9 22.4 18.0 15.8 11.4 9.1 4.2
Table 5. State-of-the-art fully-supervised methods on the AVA dataset.
Method Frame AP
AVA (with optical 
ow) [14] 15.6
ARCN (with optical 
ow) [46] 17.4
Action Transformer [11] 25.0
SlowFast (ResNet 101) [9] 26.8
SlowFast (ResNet 50, Ours) 24.9
to zero, one or many of the person tubelets at dierent keyframes in the clip,
and it is thus a more dicult task. As an AVA video clip consists of 900 seconds,
N= 900 represents the most extreme case when spatio-temporal annotations are
discarded for the entire 15-minute video.
Table 4 shows the results of our model in this setting. As expected, the
performance of our method improves the shorter the sub-clip. For N= 1 and
N= 5, our method obtains 90% and 72% of fully-supervised performance
respectively, suggesting that bounding-box level annotations are not required for
training action recognition models if the video clips are annotated over short
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid