In slightly narrower settings, prior information on the video content can allow
more ne-grained models. Articulated pose is widely used in cases where humans
are guaranteed to be present [33]. Pose estimation and activity recognition as a
joint (multi-task) problem has recently shown to improve both tasks [25]. Somewhat related to our work, Structural RNNs [19] perform activity recognition
by integrating features from semantic objects and their relationships. However,
they handle the temporal evolution of tracked objects in videos with a set of
RNNs, each of which corresponds to cliques in a graph which models the spatiotemporal relationships between these objects. This graph is hand-crafted manually for each application, though related work provides learnable connections
via gating functions [8].
Attention models are a way to structure deep networks in an often generic
way. They are able to iteratively focus attention to specic parts in the data without requiring prior knowledge about part or object positions. In activity recognition, they have gained some traction in recent years, either as soft-attention
on articulated pose (joints) [36], on feature map cells [34,39], on time [45] or on
parts in raw RGB input through dierentiable crops [3].
When raw video data is globally fed into deep neural networks, they focus
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori