have been re-implemented).
Methods Top1
R18 [16]32.05
I3D-18 [6]34.20
Ours 40.89
while solving VLOG requires detecting the right moment when the human-object
interaction occurs and thus long range reasoning plays a less important role.
Visual features extracted from object regions are the most discriminative,
however object shapes and labels also provide complementary information. Finally, the last part of Table 2 evaluates the eect of the cliques size for modeling the interactions between objects and show that pairwise cliques outperform
cliques of size 1 and 3. We would like to recall, that even with unary cliques
only, interactions between objects are still modeled. However, the model needs
to nd subspaces in the hidden representations associated to each interaction.
CNN architecture and kernel in
ations. The convolutional architecture
of the model was optimized over the validation set of the SS dataset, as shown
in Table 5. The architecture itself (in terms of numbers of layers, lters etc.)
is determined by pre-training on image classication. We optimized the choice
of lter in
ations from 2D to 2.5D or 3D for several convolutional blocks. This
has been optimized for the single head model and using a ResNet-18 variant
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori