Fig. 2. A functional overview of the model. A global convolutional model extracts
features and splits into two heads trained to predict, respectively activity classes and
object classes. The latter are predicted by pooling over object instance masks, which
are predicted by an additional convolutional model. The object instances are passed
through a visual reasoning module.
is a CNN-based architecture employing convolutional layers, including spatiotemporal convolutions, able to extract global motion features. However, it is not
able to extract information from an object level perspective. We leverage the
object head to perform reasoning on the relationships between predicted object
instances.
Our main contribution is a new structured module called Object Relation
Network (ORN), which is able to perform spatio-temporal reasoning between
detected object instances in the video. ORN is able to reason by modeling how
objects move, appear and disappear and how they interact between two frames.
In this section, we will rst describe our main contribution, the ORN network.
We then provide details about object instance features, about the activity head,
and nally about the nal recognition task. In what follows, lowercase letters
denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori