thus not trained end-to-end. Our proposed method, which is
based on the message passing neural network (MPNN) [13]
framework, coherently models both spatial and temporal
interactions. After describing our model in Sec. 3, we
show how previous structured models for action detection [54, 14, 60, 61, 71] can be thought of as special cases
of our model. Furthermore, our ﬂexible model can reason
about objects both when we have explicit supervision for
them, and when we do not. Prior work in video understanding, which have proposed less generic models, have either
assumed the case of explicit object supervision [3, 60] or
not [54, 14, 71], but have not considered the scenario when
both options are available. Furthermore, our general formulation allows us to ablate graph modelling design choices,
such as the object representation, message passing functions and temporal context, in a manner not possible with
these previous, more speciﬁc approaches. Moreover, we
note that some approaches which have employed spatiotemporal graphs in video have only considered frame-toframe interactions [22, 60, 3, 40, 42, 44], rather than longrange relations, and have not evaluated on spatio-temporal
localisation which requires more detailed understanding
3D CNN
sit, answer phone, carry/hold 
object
Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
Anurag Arnab Chen Sun Cordelia Schmid