  Convolutional Neural Networks (CNNs) have proved exceptional at learning
representations for visual object categorization. However, CNNs do not
explicitly encode objects, parts, and their physical properties, which has
limited CNNs' success on tasks that require structured understanding of visual
scenes. To overcome these limitations, we introduce the idea of Physical Scene
Graphs (PSGs), which represent scenes as hierarchical graphs, with nodes in the
hierarchy corresponding intuitively to object parts at different scales, and
edges to physical connections between parts. Bound to each node is a vector of
latent attributes that intuitively represent object properties such as surface
shape and texture. We also describe PSGNet, a network architecture that learns
to extract PSGs by reconstructing scenes through a PSG-structured bottleneck.
PSGNet augments standard CNNs by including: recurrent feedback connections to
combine low and high-level image information; graph pooling and vectorization
operations that convert spatially-uniform feature maps into object-centric
graph structures; and perceptual grouping principles to encourage the
identification of meaningful scene elements. We show that PSGNet outperforms
alternative self-supervised scene representation algorithms at scene
segmentation tasks, especially on complex real-world images, and generalizes
well to unseen object types and scene arrangements. PSGNet is also able learn
from physical motion, enhancing scene estimates even for static images. We
present a series of ablation studies illustrating the importance of each
component of the PSGNet architecture, analyses showing that learned latent
attributes capture intuitive scene properties, and illustrate the use of PSGs
for compositional scene inference.

Computer Vision and Pattern Recognition;Machine Learning