over input images and text questions, performing iterative reasoning for VQA.
While most of the discussed work has been designed for VQA and for predictions on physical systems and environments, extensions have been proposed
for video understanding. Reasoning in videos on a mask or segmentation level
has been attempted for video prediction [24], where the goal was to leverage semantic information to be able predict further into the future. Zhou et al [5] have
recently shown state-of-the-art performance on challenging datasets by extending Relation Network to video classication. Their chosen entities are frames, on
which they employ RN to reason on a temporal level only through pairwise frame
relations. The approach is promising, but restricted to temporal contextual information without an understanding on a local object level, which is provided
by our approach.
Reasoning over sets of objects is somewhat related to reasoning from unstructured data points, as done in PointNet [29], designed to learn from unordered sets
of points. PointNet shares many properties with DeepSet [46] which is a more
general framework for extracting information from sets of objects. To some extent, our work is related to PointNet, as we handle unordered sets of objects in a
permutation invariant way. However, we have an object relation viewpoint that
directly reasons over relationships between these semantic entities.
3 Object-level Visual Reasoning in Space and Time
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori