Titan XP GPUs with clips of 8 frames.
12 Baradel et al.
6 Experimental results
We evaluated the method on three standard datasets, which represent dicult
ne-grained activity recognition tasks: the Something-Something dataset, the
VLOG dataset and the recently released EPIC Kitchens dataset.
Something-Something (SS) is a recent video classication dataset with
108,000 example videos and 157 classes [13]. It shows humans performing dierent actions with dierent objects, actions and objects being combined in dierent
ways. Solving SS requires common sense reasoning and the state-of-the-art methods in activity recognition tend to fail, which makes this dataset challenging.
VLOG is a multi-label binary classication of human-object interactions
recently released with 114,000 videos and 30 classes [12]. Classes correspond
to objects, and labels of a class are 1 if a person has touched a certain object
during the video, otherwise they are 0. It has recently been shown, that stateof-the-art video based methods [6] are outperformed on VLOG by image based
methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori