modeling dierent types of interactions between items in the item set.
Note that the interactions between items in (3) are not exclusively modeled
through hc(). Indeed, it is interesting to note, that the graphical decomposition provided byCleads to interactions which are dierent from the interactions
the same decomposition would provide when used in a probabilistic graphical
model, like for instance a Markov Random Field (MRF). In particular, a decomposition into unary terms only, as given in equation (2), does notlead to
independence between items, whereas an MRF with unary terms only is equivalent to a distribution over independent random variables. This is a consequence
of the global mapping f(), which is dened on the sum over all direct interactions. Higher-order interactions between several items not directly modeled
through a non-linearity hc() can eventually be learned by the model through
the joint output space of all hc(), provided that the dimensionality Hof this
space is high enough to incorporate all interactions. However, whereas the mappinghc() provides a direct model of interactions between pairs of items, learning
interactions between two items ( j; k), which are not directly captured through a
8 Baradel et al.
visual reasoningmodule
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori