  Recently, spatiotemporal graphs have emerged as a concise and elegant manner
of representing video clips in an object-centric fashion, and have shown to be
useful for downstream tasks such as action recognition. In this work, we
investigate the use of latent spatiotemporal graphs to represent a surgical
video in terms of the constituent anatomical structures and tools and their
evolving properties over time. To build the graphs, we first predict frame-wise
graphs using a pre-trained model, then add temporal edges between nodes based
on spatial coherence and visual and semantic similarity. Unlike previous
approaches, we incorporate long-term temporal edges in our graphs to better
model the evolution of the surgical scene and increase robustness to temporary
occlusions. We also introduce a novel graph-editing module that incorporates
prior knowledge and temporal coherence to correct errors in the graph, enabling
improved downstream task performance. Using our graph representations, we
evaluate two downstream tasks, critical view of safety prediction and surgical
phase recognition, obtaining strong results that demonstrate the quality and
flexibility of the learned representations. Code is available at
github.com/CAMMA-public/SurgLatentGraph.

Computer Vision and Pattern Recognition