denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T
frames denoted by X1:T= (Xt)T
t=1where Xtis the RGB image at timestep t.
The goal is to learn a mapping from X1:Tto activity classes y.
3.1 Object Relation Network
ORN (Object Relation Network) is a module for reasoning between semantic
objects through space and time. It captures object moves, arrivals and interactions in an ecient manner. We suppose that for each frame t, we have a set
of objects kwith associated features ok
t. Objects and features are detected and
computed by the object head described in Section 3.2.
Reasoning about activities in videos is inherently temporal, as activities follow the arrow of time [28], i.e. the causality of the time dimension imposes that
past actions have consequences in the future but notvice-versa. We handle this
by sampling: running a process over time t, and for each instant t, sampling a
Object Level Visual Reasoning in Videos 7
second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=
ok
t0	K0
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori