Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we
use it for weakly-supervised spatio-temporal action recognition in Sec. 3.2. We
then describe how the standard MIL assumption, is often violated in our scenario
and describe a method to mitigate this by leveraging uncertainty estimates by
our network in Sec. 3.3. Finally, we discuss implementation details of our network
in Sec. 3.4.
3.1 Multiple Instance Learning
In the standard Multiple Instance Learning (MIL) [ 7] formulation, one is given a
bag ofNinstances, denoted as x=fx1;x2;:::;xNg. The class labels for each
of the instances is unknown, but the label for the entire bag, x, is known. The
standard MIL assumption is that a bag is assigned a class label if at least one
instance in the bag is associated with this label. More formally, we consider the
multi-label classication case, where the label vector for the bag is y2RC, and
yl= 1 if there is at least one instance with the lthlabel is present in the bag,
andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C
class labels.
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.
Anurag Arnab Chen Sun Arsha Nagrani Cordelia Schmid