2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using
the object head only) compared to an extension of Santoro et al. [32] operating
on pixel level. This indicates importance of object level reasoning. The gain on
SS is smaller (0.7 point with the full model and 7.8 points with the object head
only) and can be explained by the dierence in spatial resolution of the videos.
Object detections and predictions of the binary masks are done using the initial
video resolution. The mean video resolution for VLOG is 660 1183 and for EPIC
is 640480 against 100157 for SS. Mask-RCNN has been trained on images of
resolution 800800 and thus performs best on higher resolutions. The quality of
the object detector is important for leveraging object level understanding then
for the rest of the ablation study we focus on EPIC and VLOG datasets.
The function fin Equation (5) is an important design choice in our model.
In our proposed model, fis recurrent over time to ensure that the ORN module
captures long range reasoning over time, as shown in Equation (5). Removing
the recurrence in this equation leads to an MLP instead of a (gated) RNN, as
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori