Our approach is based on Message Passing Neural Networks (MPNN) [13], as it is a ﬂexible framework that generalises many previous graph neural network algorithms [4,
5, 29, 57] We review this approach, and describe how we
adapt it for video understanding in Sec. 3.1, before detailing our model in Sec. 3.2 through 3.5. Finally, we dis-cuss how previous structured models for video understanding [14, 54, 60, 61, 71] can be regarded as speciﬁc instantiations of our model in Sec. 3.6.
3.1. Message Passing Neural Networks (MPNNs)
MPNNs operate on a directed or undirected graph, G
consisting of nodes, v2V, and a neighbourhood for each
node,Nv, that deﬁnes the graph’s connectivity. For video
models we distinguish between the spatial, Sv, and temporal,Tv, neighbourhoods for node v(Nv=Sv[Tv). Each
node,v, is associated with a latent state, hv. Inference in
this model consists of a message passing phase, and a ﬁnal readout phase. In the message passing phase, messages
for each node, mv, are ﬁrst computed by applying spatial
Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
Anurag Arnab Chen Sun Cordelia Schmid