set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head
and object head). The intermediate feature representations UtandVtare of dimensions 2048T1414 and 2048T77, respectively. We provide a higher
spatial resolution for the feature maps Utof the object head to get more precise
local descriptors. This can be done by changing the stride of the initial conv5
layers from 2 to 1. Temporal convolutions have been congured to keep the same
time temporal dimension through the network.
Object Level Visual Reasoning in Videos 11
Global spatial pooling of activity features results in a 2048 dimensional feature vector fed into a GRU with 512 dimensional hidden state st. ROI-Pooling
of object features results in 2048 dimensional feature vectors uk
t. The encoder of
the binary mask is a MLP with one hidden layer of size 100 and outputs a mask
embedding bk
tof dimension 100. The number of object classes is 80, which leads
in total to a 2229 dimensional object feature vector ok
t.
The non-linearity h() is implemented as an MLP with 2 hidden layers each
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori