time. The ORN has potential to address these issues and conduct relational
reasoning over object interactions for the purpose of activity recognition. A set
of object detection masks ranging over dierent object categories and temporal
occurrences is input to the ORN. The ORN is able to infer pairwise relationships
between objects detected at varying dierent moments in time.
Code and object masks predictions will be publicly available6.
2 Related work
Action Recognition. Action recognition has a long history in computer vision. Pre-deep learning approaches in action recognition focused on handcrafted
spatio-temporal features including space-time interest points like SIFT-3D, HOG3D,
IDT and aggregated them using bag-of-words techniques. Some hand-crafted representations, like dense trajectories [42], still give competitive performance and
are frequently combined with deep learning.
In the recent past, work has shifted to deep learning. Early attempts adapt
2D convolutional networks to videos through temporal pooling and 3D convolutions [2,40]. 3D convolutions are now widely adopted for activity recognition with
the introduction of feature transfer by in
ating pre-trained 2D convolutional kernels from image classication models trained on ImageNet/ILSVRC [31] through
3D kernels [6]. The downside of 3D kernels is their computational complexity
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori