range from atomic human actions [14] to ne-grained object interactions [13]
to everyday, commonly occurring human-object interactions [12]. This returns
us to a human-centric viewpoint of activity recognition where it is not only the
presence of certain objects / scenes that dictate the activity present, but the
manner, order, and eects of human interaction with these scene elements that
are necessary for understanding. In a sense, this is akin to the problems in current
3D human activity recognition datasets [33], but requires the more challenging
reasoning and understanding of diverse environments common to internet video
collections.
Humans are able to infer what happened in a video given only a few sample frames. In particular, they can infer complex activities happening between
pairs of frames. This faculty is called reasoning and is a key component of human intelligence. As an example we can consider the pair of images in Figure 1,
which shows a complex situation involving articulated objects (human, carrots
and knife), the change of location and composition of objects. For humans it is
straightforward to draw a conclusion on what happened (a carrot was chopped
by the human). Humans have this extraordinary ability of performing visual reasoning on very complicated tasks while it remains unattainable for contemporary
computer vision algorithms [37,11].
The ability to perform visual reasoning in computer vision algorithms is still
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori