tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that
the object interactions are highly relevant to the detected activities: the persontouches-bed activity is correlated to interactions between relevant object classes
person and bed. Similarly, activities human-bowl interaction and human-cup interaction show interactions with the respective objects bowl and cup. Moreover,
other recovered relationships are highly correlated to the scene (for example,
dining-table and bowl for activity human-bowl interaction ).
16 Baradel et al.
Fig. 4. Example of object pairwise interactions learned by our model on VLOG for
four dierent classes. Objects co-occurrences are at the top and learned pairwise objects
interactions are at the bottom. Line thickness indicates learned importance of a given
relation. Interactions have been normalized by the object co-occurrences.
teddy bearteddy bearteddy bearbedcell phonepersonpersonbedteddy bear
bottlewine glasspersonpersonpersonpersonbottle
Fig. 5. Examples of failure cases { a) small sized objects (on the left). Our model
detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori