methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This
suggests a gap between traditional datasets like Kinetics and the ne-grained
dataset VLOG, making it particularly dicult.
EPIC Kitchens (EPIC) is an egocentric video dataset recently released
containing 55 hours recording of daily activities [7]. This is the largest in rstperson vision and the activities performed are non-scripted, which makes the
dataset very challenging and close to real world data. The dataset is densely
annotated and several tasks exist such as object detection, action recognition and
action prediction. We focus on action recognition with 39'594 action segments
in total and 125 actions classes (i.e verbs). Since the test set is not available yet
we conducted our experiments on the training set (28'561 videos). We use the
videos recorded by person 01 to person 25 for training (22'675 videos) and dene
the validation set as the remaining videos (5'886 videos).
For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori