For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,
withL=8 for the SS dataset and L=4 for VLOG and EPIC. We do not perform
any other data augmentation. While training we extract Lframes from the entire
video by splitting the video into Lsub-sequences and randomly sampling one
frame per sub-sequence. The output sequence of size Lis called a clip. A clip
aims to represent the full video with less frames. For testing we aggregate results
of 10 clips. We use lintel [10] for decoding video on the 
y.
The ablation study is done by using the train set as training data and we
report the result on the validation set. We compare against other state-of-theart approaches on the test set. For the ablation studies, we slightly decreased
the computational complexity of the model: the base network (including activity
and object heads) is a ResNet-18 instead of ResNet-50, a single clip of 4 frames
is extracted from a video at test time.
Comparison with other approaches. Table 1 shows the performance of
the proposed approach on the VLOG dataset. We outperform the state of the art
Object Level Visual Reasoning in Videos 13
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori