event occurs before the event it aects ( A!B). In Figure 1 the knife was used
before the carrot switched over to the chopped-up state on the right side. For
a video classication problem, we want to identify a causal event Ahappening
in a video that aects its label B. But instead of identifying this causal event
directly from pixels we want to identify it from an object level perspective. We
believe that such an approach would be able to learn causal signals.
Following this hypothesis we propose to make a bridge between object detection and activity recognition. Object detection allows us to extract low-level
information from a scene with all the present object instances and their semantic
meanings. However, detailed activity understanding requires reasoning over these
semantic structures, determining which objects were involved in interactions, of
what nature, and what were the results of these. To compound problems, the
semantic structure of a scene may change during a video (e.g. a new object can
appear, a person may make a move from one point to another one of the scene).
We propose an Object Relation Network (ORN), a neural network module for reasoning between detected semantic object instances through space and
time. The ORN has potential to address these issues and conduct relational
reasoning over object interactions for the purpose of activity recognition. A set
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori