performance on SS where we outperform the state of the art given by very recent
methods (+2.3 points). On EPIC we re-implement standard baselines and report
results on the validation set (Table 4) since the test set is not available. Our full
method reports an accuracy of 40.89 and outperforms baselines by a large margin
(+6.4 and+7.9 points respectively for against CNN-2D and I3D based on a
ResNet-18).
Eect of object-level reasoning. Table 2 shows the importance of reasoning on the performance of the method. The baseline corresponds to the performance obtained by the activity head trained alone (in
ated ResNet, in the
ResNet-18 version for this table). No object level reasoning is present in this
baseline. The proposed approach (third line) including an object head and the
ORN module gains 0.8, 2.5 and 2.4 points compared to our baseline respectively
on SS, on EPIC and on VLOG. This indicates that the reasoning module is able
to extract complementary features compared to the activity head.
Using semantically dened objects proved to be important and led to a gain of
2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori