computer vision algorithms [37,11].
The ability to perform visual reasoning in computer vision algorithms is still
an open problem. Attempts have been made for learning interactions between
dierent entities in images with promising results on Visual Question Answering.
There have been a number of attempts to equip neural models with reasoning
abilities by training them to solve Visual Question Answering (VQA) problems.
Among proposed solutions are prior-less data normalization [27], structuring
networks to model relationships [32,43] as well as more complex attention based
Object Level Visual Reasoning in Videos 3
mechanisms [18]. At the same time, it was shown that high performance on existing VQA datasets can be achieved by simply discovering biases in the data [21].
We extend these eorts to object level reasoning in videos . Since a video is
a temporal sequence, we leverage time as an explicit causal signal to identify
causal object relations. Our approach is related to the concept of the \arrow of
the time" [28] involving the \one-way direction" or \asymmetry" of time. Causal
event occurs before the event it aects ( A!B). In Figure 1 the knife was used
before the carrot switched over to the chopped-up state on the right side. For
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori