training neural networks to predict and control the state of articulated bodies. Ladický et al. [ 14]
recently used regression forests to learn ﬂuid dynamics. Recent advances in convolutional neural
networks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from
2
images [ 20,17,18]. Notably, Fragkiadaki et al. [ 6] used CNNs to predict and control a moving
ball from an image centered at its coordinates. Mottaghi et al. [ 21] trained CNNs to predict the 3D
trajectory of an object after an external impulse is applied. Wu et al. [ 30] used CNNs to parse objects
from images, which were then input to a physics engine that supported prediction and inference.
2 Model
Deﬁnition To describe our model, we use physical reasoning as an example (Fig. 1a), and build
from a simple model to the full interaction network (abbreviated IN). To predict the dynamics of a
single object, one might use an object-centric function, fO, which inputs the object’s state, ot, at
timet, and outputs a future state, ot+1. If two or more objects are governed by the same dynamics,
fOcould be applied to each, independently, to predict their respective future states. But if the
Reasoning about objects, relations, and physics is central to human
intelligence, and a key goal of artificial intelligence. Here we introduce the
interaction network, a model which can reason about how objects in complex
systems interact, supporting dynamical predictions, as well as inferences about
the abstract properties of the system. Our model takes graphs as input,
performs object- and relation-centric reasoning in a way that is analogous to a
simulation, and is implemented using deep neural networks. We evaluate its
ability to reason about several challenging physical domains: n-body problems,
rigid-body collision, and non-rigid dynamics. Our results show it can be
trained to accurately simulate the physical trajectories of dozens of objects
over thousands of time steps, estimate abstract quantities such as energy, and
generalize automatically to systems with different numbers and configurations
of objects and relations. Our interaction network implementation is the first
general-purpose, learnable physics engine, and a powerful general framework for
reasoning about object and relations in a wide variety of complex real-world
domains.
Peter W. Battaglia Razvan Pascanu Matthew Lai Danilo Rezende Koray Kavukcuoglu