Fig. 3. ORN in the object head operating on detected instances of objects.
clique cand its corresponding hc(), requires learning a corresponding subspace
in the common output space spanned by all hc().
This leads to the question of how to dene the trade-o between the complexity of the decomposition Cand the output dimensionality Hof the mapping
hc(), both of which will determine the complexity of the modeled interactions.
Increasing the size of cliques in Cwill increase the input dimension (and therefore the capacity) of the mapping hc() as well as the computational complexity
of the sum operation.
Inspired by relational networks [32], we chose to directly model inter-frame
interactions between pairs of objects ( j; k) and leave modeling of higher-order
interactions to the output space of the mappings hand the global mapping f:
gt=X
j;kh(oj
t0;ok
t) (4)
In order to better directly model long-range interactions, we make the global
mapping f(;) recurrent, which leads to the following form:
rt=f(gt;rt1) (5)
where rtrepresents the recurrent object reasoning state at time tandgtis the
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori