datasets [6,26,13,12,23,14], which allow the training of high-capacity deep learning models from massive amounts of data. These models enable detection of key
cues present in videos, such as global and local motion, various object categories
and global scene-level information, and often achieve impressive performance in
recognizing high-level, abstract concepts in the wild.arXiv:1806.06157v3  [cs.CV]  20 Sep 2018
2 Baradel et al.
person 0.86person 0.87knife 0.98carrot 0.99carrot 0.72carrot 0.98
person 0.65knife 0.99carrot 0.97carrot 0.97carrot 0.93carrot 0.86carrot 0.95
Fig. 1. Humans can understand what happened in a video (\the leftmost carrot was
chopped by the person") given only a pair of frames. Along these lines, the goal of this
work is to explore the capabilities of higher-level reasoning in neural models operating
at the semantic level of objects and interactions.
However, recent attention has been directed toward a more thorough understanding of human-focused activity in diverse internet videos. These eorts
range from atomic human actions [14] to ne-grained object interactions [13]
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori