state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object
reasoning state output by the ORN module. We perform independent linear
classication for each representation:
y1=W h (8)
y2=Z r (9)
where y1;y2correspond to the logits from the activity head and the object head ,
respectively, and WandZare trainable weights (including biases). The nal
prediction is done by averaging logits y1andy2followed by softmax activation.
4 Network Architectures and feature dimensions
The input RGB images Xtare of size R3WHwhere WandHcorrespond to
the width and height and are of size 224 each. The object and activity heads
(orange and green in Figure 2) are a joint convolutional neural network with
Resnet50 architecture pre-trained on ImageNet/ILSVRC [31], with Conv1 and
Conv5 blocks being in
ated to 2.5D convolutions [44] (3D convolutions with a
separable temporal dimension). This choice has been optimized on the validation
set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori