a plug-in module for reasoning in deep networks. RN shows human-level performance in Visual Question Answering (VQA) by inferring pairwise \object"
relations. However, in contrast to our work, the term \object" in [32] does not
refer to semantically meaningful entities, but to discrete cells in feature maps.
The number of interactions therefore grows with feature map resolutions, which
makes it dicult to scale. Furthermore, a recent study [21] has shown that some
of these results are subject to dataset bias and do not generalize well to small
changes in the settings of the dataset.
In the same line, a recent work [38] has shown promising results on discovering objects and their interactions in an unsupervised manner using training
examples from virtual environments. In [41], attention and relational modules
are combined on a graph structure. From a dierent perspective, [27] show that
relational reasoning can be learned for visual reasoning in a data driven way without any prior using conditional batch normalization with a feature-wise ane
transformation based on conditioning information. In an opposite approach, a
strong structural prior is learned in the form of a complex attention mechanism: in [18], an external memory module combined with attention processes
over input images and text questions, performing iterative reasoning for VQA.
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori