entire video content in addition to relational reasoning for making ne distinctions regarding object interactions and properties.
In our method, local low-level reasoning is provided through object head
and the ORN module such as described above in Section 3.1. We complement
10 Baradel et al.
this representation by high-level context information described by Vtwhich are
feature outputs from the activity head (orange block in Figure 2).
We use spatial global average pooling over Vtto output T D-dimensional
feature vectors denoted by vt, where vtcorresponds to the context information
of the video at timestep t.
We model the dynamics of the context information through time by employing a RNN f
() given by:
st=f
(vt;st1) (7)
where sis the hidden state of f
() and gives cues about the evolution of the
context though time.
3.4 Recognition
Given an input video sequence X1:T, the two dierent streams corresponding to
the activity head and the object head result in the two representations handr,
respectively where h=P
thtandr=P
trt. Each representation is the hidden
state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori