3D kernels [6]. The downside of 3D kernels is their computational complexity
6https://github.com/fabienbaradel/object_level_visual_reasoning
4 Baradel et al.
and the large number of learnable parameters, leading to the introduction of
2.5D kernels, i.e. separable lters in the form of a 2D spatial kernel followed by
a temporal kernel [44]. An alternative to temporal convolutions are Recurrent
Neural Networks (RNNs) in their various gated forms (GRUs, LSTMs) [17,9].
Karpathy et al. [20] presented a wide study on dierent ways of connecting
information in spatial and temporal dimensions through convolutions and pooling. On very general datasets with coarse activity classes they have showed that
there was a small margin between classifying individual frames and classifying
videos with more sophisticated temporal aggregation.
Simoyan et al. [35] proposed a widely adopted two-stream architecture for
action recognition which extracts two dierent streams, one processing raw RGB
input and one processing pre-computed optical 
ow images. The method outperformed the state of the art, but relies on rather small scale optical 
ow computations.
In slightly narrower settings, prior information on the video content can allow
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori