is required at a very low level (motion estimation) as well as on a very high
level, close to reasoning. Our study also corroborates recent research in activity
recognition, indicating that 2.5D kernels provide a good trade-o between highcapacity and learnable numbers of parameters. Finally temporal integration via
RNN outperforms global average pooling over space and time. The choice of
a (gated) RNN for temporal integration of the activity head features proved
important (see Table 5) compared to global average pooling (GAP) over space
and time.
Visualizing the learned object interactions. Figure 4 shows visualizations of the pairwise object relationships the model learned from data, in particular from the VLOG dataset. Each graph is computed for a given activity class,
and strong arcs between two nodes in the graph indicate strong relationships
between the object classes, i.e. the model detects a high correlation between
these relationships and the activity. The graphs were obtained by thresholding
the summed activations of each pairwise relationship ( j; k) in equation (4). Each
pair ( j; k) can be assigned a pair of object classes cj
tandck
tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori