localisation which requires more detailed understanding
3D CNN
sit, answer phone, carry/hold 
object
sit, touch (an object), watch (a 
person), talk (a person)
sit, touch (an object), watch (a 
person), listen (a person)
Readout Function, 

Spatio
temporal graph 
construction and message passing
Foreground node
Implicit context
Explicit contextFigure 2: Overview of our method: We construct a spatio-temporal graph, and perform message-passing inference on it, to model interactions between actors, objects and their environment. Foreground nodes (circles) have readout functions associated with them for the task of
interest ( i.e. for action recognition, the nodes represent person bounding boxes which are classiﬁed into actions). Context nodes (squares)
model additional information, and can either be implicit, as cells of the original feature map, or explicit by ROI-pooling external region
proposals (RPN [46] not shown for clarity). The initial state of each node is a spatio-temporal feature vector extracted from a 3D CNN.
and is essential for analysing untrimmed videos.
We also note that scene graph parsing [25, 30] is another task that evaluates a model’s ability to reason about
Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
Anurag Arnab Chen Sun Cordelia Schmid