and is essential for analysing untrimmed videos.
We also note that scene graph parsing [25, 30] is another task that evaluates a model’s ability to reason about
the interactions between different objects by representing
objects as nodes and relationships as edges in a graph. Although the task was originally posed for single images [30],
the recent Action Genome [23] dataset extends this task
to video by adding annotations to Charades [49]. While
GNN-based approaches have also been used in scene graph
parsing [10, 33, 64, 66] for single images, to our knowledge, they have not been applied to model temporal relations in video. Moreover, the same model has not also been
demonstrated on spatio-temporal action recognition like our
method.
3. Proposed Approach
Our model aims to build a structured representation of
a video by representing it as a graph of actors, objects
and contextual elements in the scene, as shown in Fig. 2.
This structured representation is then used to perform tasks
which require understanding the interactions between the
elements in the graph, such as action recognition and scene
graph prediction. Note that we do not assume we have annotations for relevant scene context.
Our approach is based on Message Passing Neural Networks (MPNN) [13], as it is a ﬂexible framework that generalises many previous graph neural network algorithms [4,
Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
Anurag Arnab Chen Sun Cordelia Schmid