LEARNING PHYSICAL DYNAMICS
We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE’s compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.
1 INTRODUCTION
Endowing an agent with a program for physical reasoning constrains the agent’s representation of the environment by establishing a prior on the environment’s physics. The agent can leverage these constraints to rapidly learn new tasks, to flexibly adapt to changes in inputs and goals, and to naturally generalize reasoning to novel scenes (Lake et al., 2016).
For example, a foundational sense of intuitive physics is a prior that guides humans to decompose a scene into objects and carry expectations of object boundaries and motion across different scenarios (Spelke, 1990). Humans perceive balls on a billiard table not as meaningless patches of color but rather as impermeable objects. They expect balls moving toward each other to bounce a certain way after a collision rather than pass through each other, crumble into pieces, or disperse into smoke. Replace one billiard ball with a bowling ball and expectations for ball-to-ball interactions will differ, but the underlying sense of inertia and collisions remain. Arrange immovable wooden obstacles on the table and expectations for how a ball’s surface interacts with wood remain constant regardless of how the obstacles are arranged. The ability to plan trajectories in this space without having to relearn physics from scratch each time, regardless of whether there are three balls or eight balls, whether there are obstacles or not, whether obstacles are arranged in one way or another, whether or not the configuration of objects has been seen before, suggests that humans leverage a prior on physics to reason at a level of abstraction where objects, relations, and events are primitive.
This paper explores the question of building this prior into an agent as a program. We view this program as a simulator that takes input provided by a physical scene and the past states of objects, and outputs the future states and physical properties of relevant objects (Anderson, 1990; Battaglia et al., 2013; Goodman and Tenenbaum, 2016). Our goal is to design a program that naturally generalizes across variable object count and different scene configurations without additional retraining. Our proposed framework, the Neural Physics Engine (NPE), outlines several ingredients useful for realizing these two generalization capabilities. We describe these ingredients in the context of a specific instantiation of the NPE applied to two-dimensional worlds of balls and obstacles.
1.1 A HYBRID DESIGN
Two general approaches have emerged in the search for a program that captures common-sense physical reasoning. The top-down approach (Bates et al., 2015; Battaglia et al., 2013; Hamrick et al., 2011; Ullman et al., 2014; Wu et al., 2015) formulates the problem as inference over the parameters of a symbolic physics engine, while the bottom-up approach (Agrawal et al., 2016; Fragkiadaki et al., 2015b; Lerer et al., 2016; Li et al., 2016; Mottaghi et al., 2015; 2016; Sutskever et al., 2009) learns to directly map observations to motion prediction or physical judgments. A program under the top-down approach can generalize across any scenario supported by the entities and operators in its description language. However, it may be brittle under scenarios not supported by its description language, and adapting to these new scenarios requires modifying the code or generating new code for the physics engine itself. In contrast, gradient-based bottom-up approaches can apply the same model architecture and learning algorithm to specific scenarios without requiring the physical dynamics of the scenario to be pre-specified. This often comes at the cost of reduced generality: transferring knowledge to new scenes may require extensive retraining, even in cases that seem trivial to human reasoning.
The NPE takes a step toward bridging the gap between expressivity and adaptability by combining the strengths of both approaches. The NPE framework is realized as a differentiable physics simulator that combines rough symbolic structure with gradient-based learning. It exhibits several strong inductive biases that are explicitly present in symbolic physics engines, such as a notion of objects-specific properties and object interactions. Implemented as a neural network, the NPE can also flexibly tailor itself to specific object properties and dynamics of a given world through training. By design, it can extrapolate to a variable number of objects and different scene configurations with only spatially and temporally local computation.
1.2 INGREDIENTS USEFUL FOR GENERALIZATION
Our framework proposes four key ingredients useful for generalization across variable object count and different scene configurations without additional retraining. The first ingredient is the view of objects as primitives of physical reasoning. The second is a mechanism for selecting context objects given a particular object. Together, these ingredients reflect two natural assumptions about a physical environment: There exist objects and these objects interact in a factorized manner.
The third and fourth ingredients are factorization and compositionality, which are both applied on two levels: the scene and the network architecture. On the level of the physical scene, the NPE factorizes the scene into object-based representations, and composes smaller building blocks to form larger objects. This method of representation adapts to scene configurations of variable complexity and shape. On the level of the network architecture, the NPE explicitly reflects a causal structure in object interactions by factorizing object dynamics into pairwise interactions. The NPE models the future state of a single object as a function composition of the pairwise interactions between itself and other context objects in the scene. This structure serves to guide learning towards objectbased reasoning and is designed for physical knowledge to transfer across variable number objects anywhere in the scene.
1.3 A STEP TOWARDS EMULATING A GENERAL-PURPOSE PHYSICS ENGINE
While previous bottom-up approaches (Sec. 4) have coupled learning vision and learning physical dynamics, we take a different approach for two reasons. First, we see that disentangling the visual properties of an object from its physical dynamics is a step toward achieving the generality of a physics engine. Both vision and dynamics are necessary, but we believe that keeping these functionalities separate is important for common-sense generalization that is robust to cases where the visual appearance changes but the dynamics remain the same. Second, we are optimistic that those two components indeed can be decoupled, that a vision model can map visual input to an intermediate state space, and a dynamics model can evolve objects in that state space through time. For example, there is work in object detection and localization (e.g. Eslami et al. 2016) for extracting position and velocity, as well as work for extracting latent object properties (Wu et al., 2015; 2016). Therefore this paper focuses on learning dynamics in that state space, taking a small step toward emulating a general-purpose physics engine, with the eventual goal of building a system that exhibits the compo-
sitionality, modularity, and generality of a physics engine whose internal components can be learned through observation.
In Sec. 2 we present a specific instantiation of the NPE that uses a neighborhood mask to select context objects. In Sec. 3 we apply that instantiation to investigate variations on two-dimensional worlds of balls and obstacles from the matter-js physics engine (Brummitt, 2014) as a testbed for exploring the NPE’s capabilities to model simple rigid-body dynamics. While these worlds are generated from a simplified physics engine, we believe that learning to model such simple physics under the NPE’s framework is a first and necessary step towards emulating the full capacity of a general physics engine, while maintaining a differentiability that can allow it to eventually learn complex real-world physical phenomena that would be challenging to engineer into conventional physics engines. This paper establishes that important step.
2 APPROACH
2.1 NEURAL PHYSICS ENGINE
We consider in detail a specific instantiation of the NPE that uses a neighborhood mask to select context objects. This section discusses each of the four ingredients of the NPE framework, that, when combined, comprise a neural network-based physics simulator that learns from observation.
Object-Based Representations We make two observations (Fig. 1) in our factorization of the scene. The first regards spatially local computation. Because physics does not change across inertial frames, it suffices to separately predict the future state of each object conditioned on the past states of itself and the other objects in its neighborhood, similar to Fragkiadaki et al. (2015b). Sec. 3.5 shows that when large structures are represented as a composition of smaller objects, a spatially local attention window helps achieve invariance to scene configuration. The second observation regards temporally local computation. Because physics is Markovian, this prediction need only be for the immediate next timestep, which we show in Sec. 3 is enough to predict physics effectively over long timescales. Given these two observations, it is natural to choose an object-based state representation. A state vector comprises extrinsic properties (position, velocity, orientation, angular velocity), intrinsic properties (mass, object type, object size), and global properties (gravitational, frictional, and pairwise forces) at a given time instance.
Pairwise Factorization Letting a particular object be the focus object f and all other objects in the scene be context objects c, the NPE models the focus object’s velocity v[t+1]f as a composition of the pairwise interactions between itself and other neighboring context objects in the scene during time t − 1 and t. This input is represented as pairs of object state vectors {(of , oc1)[t−1,t], (of , oc2)[t−1,t], ...}. As shown in Fig. 2b, the NPE composes an encoder function and a decoder function. The encoder function fenc summarizes the interaction of a single object pair. The sum of encodings of all pairs is then concatenated with the focus object’s past state as input to the decoder function. The focus object is a necessary input to the decoder because if there are no neighboring context objects, the summed encoder output would be zero. The decoder function then predicts the focus object’s velocity v[t+1]f . In practice, the NPE predicts the change ∆v between t
and t + 1 to compute v[t+1] = v[t] + ∆v, and updates position using the velocity as a first-order approximation1. We predict velocity rather than position to help avoid memorizing the environment; training the network to predict position conditions the network on the worlds in the training domain, making it more difficult to transfer knowledge across environments. We do not include acceleration in the state representation because position and velocity fully parametrize an object’s state. Thus acceleration (e.g. collisions) can be learned by observing velocity for two consecutive timesteps, hence our choice for two input timesteps. We explored longer input durations as well and found no additional benefit.
Context Selection Each (of , oc) pair is selected to be in the set of neighbors of f by the neighborhood masking function 1 [||pc − pf )|| < N(of )], which takes value 1 if the Euclidean distance between the positions pf and pc of the focus and context object respectively at time t is less the neighborhood threshold N(of ). Many physics engines use a collision detection scheme with two phases. Broad phase is used for computational efficiency and uses a neighborhood threshold to select objects that might, but not necessarily will, collide an object. Narrow phase performs the actual collision detection on that smaller subset of objects and also resolves the collisions for the objects that do collide. Analogously, our neighborhood mask implements broad phase, and the NPE implements narrow phase. The mask only constrains the search space of context objects, and the network figures out how to detect and resolve collisions. This mask is a specific case of a more general attention mechanism to select contextual elements of a scene.
Function Composition Symbolic physics engines evolve objects through time based on dynamics that dictate their independent behavior (e.g. inertia) and their behavior with other objects (e.g. collisions). Notably, in a particular object’s reference frame, the forces it feels from other objects are additive. The NPE architecture incorporates several inductive biases that reflect this recipe. The composition of fenc and fdec induce a causal structure on the pairs of objects. We provide a loose interpretation of the encoder output ec,f as the effect of object c on object f , and require that these effects are additive as forces are. This design allows the NPE to scale naturally to different numbers of neighboring context objects. These inductive biases have the effect of strongly constraining the space of possible simulators that the NPE can learn, focusing on compositional programs that reflect pairwise causal structure in object interactions.
2.2 BASELINES
The purpose of contrasting the NPE with the following two baselines is to illustrate the benefit of pairwise factorization and function composition, which are the key architectural features of the NPE. As the architectures for both baselines have been shown to work well in similar tasks, it is not immediately clear whether the NPE’s assumptions are useful or necessary, so these are good baselines for comparison. Viewed in another way, comparing with these baselines is a lesion study on the NPE because each baseline lacks an aspect of the NPE structure.
No-Pairwise The No-Pairwise (NP) baseline is summarized by Fig. 2c. It is very similar to the NPE but does not compute pairwise interactions; otherwise its encoder and decoder are the same as the NPE’s. Therefore the NP most directly highlights the value of the NPE’s pairwise factorization. The NP is also a Markovian variant of the Social LSTM (Alahi et al., 2016); it sums the encodings of context objects after encoding each object independently, similar to the Social LSTM’s “social pooling.” Information for modeling how objects interact would only be present after the encoding step. A possible mechanism for predicting dynamics with the NP is if the encoder’s object encoding consists of an abstract object representation and a force field created by that object. Therefore the decoder could apply the sum of the force fields of all context objects to the focus object’s abstract object representation to predict the focus object’s velocity. As Alahi et al. (2016) has demonstrated the Social LSTM’s performance in modeling human trajectories, it would be interesting to see how the same architectural assumptions perform for the physics of moving objects.
1The NPE as currently implemented also predicts angular velocity along with velocity, but for the experiments in this paper we always set angular velocity, as well as gravity, friction, and pairwise forces, to zero. We included these parameters in the implementation because in future work we are planning to test situations and scenarios in which angular velocity is important, such as block towers, magnetism. However, in the current work they are vestigial and set to zero and do not appear in the evaluation.
LSTM Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) have been shown to sequentially attend to objects (Eslami et al., 2016), so it is interesting to test whether a LSTM is well-suited for modeling object interactions, when the object states are explicitly given as input. From a cognitive science viewpoint, an LSTM can be interpreted as a serial mechanism in object tracking (Pylyshyn and Annan, 2006). Our LSTM architecture (Fig. 2d) accepts the state of each context object until the last step, at which it takes in the focus object’s state and predicts its velocity. Because the LSTM moves through the object space sequentially, its lack of factorized compositional structure highlights the value of the NPE’s function composition of the independent interactions between an object and its neighbors. Our notion of compositionality treats each object and pairwise interaction as independently encapsulated in a separate computational entity that can be reused and rearranged; the NPE encoder is a function that is applied to each (of , oc) pair. This function encapsulates this computation and can be repeatedly applied to all neighboring context objects equally, such that the NPE composes this repeated encoding function with the decoder function to predict velocity. The LSTM does not exhibit this notion of compositionality because it is not designed to take advantage of the factorized structure of the scene. Unlike the NPE and NP, the LSTM’s structure does not differentiate between focus and context object, so we add a flag to the state representation to indicate to whether an object is a context or focus object. We shuffle the order of the context objects to account for an ordering bias.
3 EXPERIMENTS
Object-based representations (ingredient 1) are necessary for the other three ingredients, and having explained the motivation for object-based representations in Sec. 1.3 and Sec. 2.1, we now analyze the other three ingredients in the context of several experiments. In the prediction task (Sec. 3.1), we first test if the NPE is even capable of predicting physics when the number of objects is held constant. In the generalization task (Sec. 3.2), we test the NPE’s capability to generalize across variable object count. In the inference task (Sec. 3.3), we test if the NPE can be inverted to infer mass in both the prediction and generalization settings. In these experiments, we compare against the NPE-NN, a modified NPE without the neighborhood mask, to analyze the context selection
mechanism (ingredient 2), the NP to analyze factorization (ingredient 3), the LSTM to analyze compositionality (ingredient 4). Sec. 3.4 analyzes the neighborhood mask in depth. We test the NPE’s capability to generalize across different scene configurations in Sec. 3.5.
Using the matter-js physics engine, we evaluate the NPE on worlds of balls and obstacles. These worlds exhibit nonlinear dynamics and support a wide variety of scenarios. Bouncing balls have been of interest in cognitive science to study causality and counterfactual reasoning, as in Gerstenberg et al. (2012). We trained on 3-timestep windows in trajectories of 60 timesteps (10 timesteps ≈ 1 second). For a world of k objects, we generate 50,000 such trajectories. For experiments where we train on multiple worlds together, we shuffle the examples across all training worlds and train without a curriculum schedule. All worlds have a vertical dimension of 600 pixels and a horizontal dimension of 800 pixels, and we constrain the maximum velocity of an object to be 60 pixels/second. We normalize positions to [0, 1] by dividing by the horizontal dimension, and we normalize velocities to [−1, 1] by dividing by the maximum velocity. Like those of Battaglia et al. (2016) the NPE predictions can be effective over long timescales even when the NPE is only trained to predict the immediate next time step. Randomly selected simulation videos can be found at https://goo.gl/BWYuOF. Plots show results over three independent runs averaged over held-out test data with different random seeds. As shown in the graphs in Fig. 3 (top two rows) and Fig. 5, both the NP and LSTM’s predicted trajectories diverge from the ground truth, but for different reasons, which the videos illuminate. While the NP and LSTM fail to predict plausible physical movement entirely, the NPE’s predictions initially adhere closely to the ground truth, then slowly diverge due to the accumulation of subtle errors, just as the human perceptual system also accumulates errors (Smith and Vul, 2013). However, the NPE preserves the general intuitive physical dynamics that may roughly be consistent with people’s intuitive expectations.
3.1 PREDICTION TASK
We consider simple worlds of four balls of uniform mass (Fig. 3a). To measure performance in simulation, we visualize the cosine similarity between the predicted velocity and the ground truth velocity as well as the relative error in magnitude between the predicted velocity and the ground truth velocity over 50 timesteps of simulation. The models take timesteps 1 and 2 as initial input, and then use previous predictions as input to future predictions. To measure progress through training, we also display the Mean Squared Error (MSE) on the normalized velocity.
3.2 GENERALIZATION TASK
We test whether learned knowledge of these simple physics concepts can be transferred and extrapolated to worlds with a number of objects previously unseen (Fig. 3b). The unseen worlds (6, 7, 8 balls) in the test data are combinatorially more complex and varied than the observed worlds (3, 4, 5 balls) in the training data. All objects have equal mass. During simulation, the NPE’s predictions are more consistent, whereas the NP and LSTM’s prediction begin to diverge wildly towards the end of 50 timesteps of simulation (Fig. 3b, middle row). The NPE consistently outperforms the baselines by 0.5 to 1 order of magnitude in velocity prediction (Fig. 3b, bottom row).
3.3 INFERENCE TASK
We now show that the NPE can infer latent properties such as mass. This proposal is motivated by the experiments in Battaglia et al. (2013), which uses a probabilistic physics simulator to infer various properties of a scene configuration. Whereas the physical rules of their simulator were manually pre-specified, the NPE learns these rules from observation. We train on the same worlds used in both the prediction and generalization tasks, but we uniformly sampled the mass for each ball from the log-spaced set {1, 5, 25}. We chose to use discrete-valued masses to simplify our qualitative understanding of the model’s capacity to infer. For future work we would like to investigate continuously valued masses and evaluate with binary comparisons (e.g. ”Which is heavier?”).
As summarized by Fig. 3c and Fig. 4a, we select scenarios exhibiting collisions with the focus object, fix the masses of all other objects, and score the NPE’s prediction under all possible mass hypotheses for the focus object. The prediction is scored against the ground-truth under the same MSE loss used in training. The hypothesis whose prediction yields the lowest error is the NPE’s
maximum likelihood estimate of the focus object’s mass. Outperforming all baselines, the NPE achieves about 90% accuracy, meaning it has 90% probability of inferring the correct mass.
The NPE predicts outputs given inputs and infers inputs given outputs. Though we adopted a particular parametrization of an object, the NPE is not limited to the semantic meaning of the elements of its input, so we expect other latent object properties can be inferred this way. Because the NPE is differentiable, we expect that it can also infer object properties by backpropagating prediction error to its a randomly sampled input. This would be useful for inferring non-categorical values, such as positions of “invisible” objects, whose effects are felt but whose positions are unknown.
3.4 NEIGHBORHOOD MASK
In Fig. 3d we vary the NPE’s neighborhood threshold N(of ) and evaluate performance on the constant-mass prediction task. N(of ) is in units of ball radii, so N(of ) = 2 means that a context object is only detected if it is exactly touching the focus object. Because ball radii are 60 pixels and the maximum velocity is 60 pixels per timestep, the maximum distance two balls can initially be before touching at the next timestep is 4 ball radii. Given that velocities were sampled uniformly, it makes sense that the NPE performs well in and is robust2 to the range N(of ) ∈ [3, 5], but performance drops off with smaller and larger N(of ). It is important to note that different N(of ) may work better for different domains and object geometries.
2The results reported in this paper were with N(of ) = 3.5 ball radii, which we found initially with a coarser search than the results in Fig. 3d, although any threshold in the range N(of ) ∈ [3, 5] performs similarly.
We include analysis in the prediction and generalization tasks on an NPE without the neighborhood mask, the NPE-NN (NN = No Neighborhood). The neighborhood mask gives the NPE about an order of magnitude improvement in velocity prediction loss (Fig. 3a,b: bottom row and Fig. 6). While the NPE loss continues to improve through training, the NPE-NN loss quickly plateaus. It is interesting that the NPE-NN performs no better than both the NP and LSTM in predictive error, but outperforms the LSTM in mass inference. These two observations suggest that computing the interactions the focus object shares with each context object is more effective for inferring a property of the focus object than disregarding these factorized effects. They also suggest that the additional spatial structure from constraining the context space with the neighborhood mask prevents the NPE from naively finding associations with objects that cannot influence the focus object.
In our experiments, the neighborhood mask has the additional practical benefit of reducing computational complexity from O(k) to O(1), where k is the number of objects in the scene, because the number of context-focus object pairs the NPE considers is bounded above by the neighborhood mask at a constant number. Though beyond the scope of this work, to extend the functionality of such context selection mechanism to include worlds that contain forces that act from a distance, future instantiations of the NPE may investigate a more general context selection mechanism that can be learned jointly with the other model parameters.
3.5 DIFFERENT SCENE CONFIGURATIONS
We demonstrate representing large structures as a composition of smaller objects as building blocks. This is important for testing the NPE’s invariance to scene configuration; the scene configuration should not matter if the underlying physical laws remain the same. These worlds contain 2 balls bouncing around in variations of 4 different wall geometries. “O” and “L” geometries have no internal obstacles and are in the shape of a rectangle and “L” respectively. “U” and “I” have internal obstacles. Obstacles in “U” are linearly attached to the wall like a protrusion, while obstacles in “I”
have no constraint in position. We randomly vary the position and orientation of the “L” concavity and the “U” protrusion. We randomly sample the positions of the “I” internal obstacles.
We train on conceptually simpler “O” and “L” worlds and test on more complex “U” and “I” worlds. Variations in wall geometries adds to the difficulty of this extrapolation task. At most 12 context objects are present in the focus object’s neighborhood at a time. The “U” geometries have 33 objects in the scene, the most out of all the wall geometries. As shown in Fig. 4b and 5, the NPE is robust to scenes with internal obstacles, even when it has not observed such scenes during training.
3.6 ANALYSIS
We explain the NPE’s superior performance in generalization from the perspective of context selection, factorization, and compositionality. By design, all three ingredients transform the testing data distribution to be similar to the training data distribution, such that generalization across variable object count and different scene configurations happens naturally.
Consider generalizing across variable object count. The neighborhood mask selects context objects such that the NPE need only focus on a bounded subset of the objects regardless of the total number of objects. Factorizing the scene into pairwise interactions induces a causal structure between each context object and the focus object, such that no matter the object count, this causal structure remains consistent because the input is merely a set of object pairs. Composing these pairwise interactions together with a summation encourages the encoder output to be additive, such that the decoder receives the appropriate net effect from the context objects, regardless of how many there are.
Consider generalizing across different scene configurations. Our state representation composes larger structures from smaller objects, just as many real-world objects are composed of smaller components. Therefore, even when wall geometries are complex and varied on a macroscopic scale, the input distribution to the NPE remains roughly the same, because the prediction problem still remains only over objects in a local glimpse the entire scene.
4 RELATED WORK
Top-down and bottom-up approaches A recent set of top-down approaches investigate probabilistic game physics engines as computational models for physical simulation in humans (Bates et al., 2015; Battaglia et al., 2013; Hamrick et al., 2011; Ullman et al., 2014). However, these models
require a full specification of the physical laws and object geometries. Given such a specification, inferring how physical laws compose and apply to a given scenario are their strength, but automatically inferring from visual data what physical laws and object properties are present requires more work in inverse graphics (Chen et al., 2016; Kulkarni et al., 2014; 2015a;b; Whitney et al., 2016) and physics-based visual understanding (Brand, 1997; Wu et al., 2015; 2016). The NPE builds on top of the key structural assumptions of these top-down approaches, but its differentiable architecture opens a possible path for joint training with a vision model that can automatically adapt to the specific physical properties of the scene.
Bottom-up approaches attempt to bypass the intermediate step of finding physics representations and directly map visual observations to physical judgments (Lerer et al., 2016; Li et al., 2016; Mottaghi et al., 2015; 2016) or passive (Lerer et al., 2016; Srivastava et al., 2015; Sutskever et al., 2009) and action-conditioned (Agrawal et al., 2016; Finn et al., 2016; Fragkiadaki et al., 2015b) motion prediction. Because these work historically have not been compositional in nature, they have had limited flexibility to transfer knowledge to conceptually similar worlds where the physics remain the same, but the number of objects or complexity of object configurations varies. Moreover, these approaches above do not infer latent properties as the NPE does.
Other work have taken similar hybrid approaches as the NPE, such as the NeuroAnimator (Grzeszczuk et al., 1998), one of the first work to train a neural network to emulate a physics simulator, and the interaction network (Battaglia et al., 2016), which learns to simulate physics over a graph of objects and their relations.
Sketching The NPE combines a symbolic structure that assumes generic objects and interactions with a differentiability that allows the specific nature of these interactions to be learned from training. This approach of starting with a general sketch of a program and filling in the specifics is inspired by ideas from the program synthesis community (Ellis et al., 2015; Gaunt et al., 2016; Solar-Lezama, 2008). Examples of other work that combine symbolic with neural approaches via sketching include graph-based neural networks (Jain et al., 2016; Li et al., 2015; Scarselli et al., 2009) and transforming autoencoders (Hinton et al., 2011).
Composing functions for reuse Just as the NPE repeatedly applies the same encoder to each object pair, iteratively applies itself to each object in the scene as a focus object, and recursively predicts future timesteps using predictions from previous timesteps, employing function reuse to achieve generalization is also featured in work such as Abelson et al. (1996); Andreas et al. (2016); Lake et al. (2015); Reed and de Freitas (2015); Socher et al. (2011). These work all assemble small subprograms to form larger programs. The NPE also dynamically composes its internal modules (encoder and decoder) based on the number of objects and the arrangement of context objects.
Object-based approaches Fragkiadaki et al. (2015b) and Battaglia et al. (2016) are two notably similar work in the sense that our work and theirs all take an object-based approach to model the bouncing balls environment. Our work was inspired by Fragkiadaki et al. (2015b)’s iterative approach to predicting the motion of each object in turn, conditioned on a context. The key contrast is that their model assumes no relational structure between objects beyond a visual attention window centered around the focus object, whereas ours explicitly processes the interaction between the focus and each context object.
If we compare their simulation videos (Fragkiadaki et al., 2015a) to ours, we see some specific and significant improvements evident in our approach. For example, in their work, the balls appear attracted to each other and to the walls; the balls appear to bounce along the walls even when no attractive force should be present. The balls rarely touch during collisions, but magnetically repel each other when at a short distance. The NPE does not exhibit these behaviors and tends to preserve the intuitive physical dynamics of colliding balls. In addition to these differences, we show strong predictive performance on generalizing to eight balls, five more than the balls in their videos. We also crucially show this performance under stronger generalization conditions, variable mass, and more complex scene configurations.
Recently, Battaglia et al. (2016) independently and in parallel developed an architecture that they call the interaction network for learning to model physical systems. They show how such an architecture can apply to several different kinds of physical systems, including n-body gravitational interactions
and a string falling under gravity. Like their work, our model can simulate over many timesteps very effectively when only trained for next-timestep prediction, and can generalize to different world configurations and different numbers of objects.
Compared to the interaction network, a main difference in our architecture is that ours does not take object relations as explicit input, but instead learns the nature of these relations by constraining attention to a neighborhood set of objects. Another difference is in function reuse: we demonstrated that a trained NPE can automatically infer properties of its input such as mass without further retraining. In contrast, they train an additional classifier on top of their model to do inference. Their work also exhibits the four ingredients in our framework, and we view the similarities between their and our work as converging evidence for the utility of object-based representations and compositional model architectures in learning to emulate general-purpose physics engines.
5 DISCUSSION
While this paper is not the first to explore learning a physics simulator, here we take the opportunity to highlight the value of this paper’s contributions. We hope these contributions can seed further research that builds on the NPE framework this paper proposes.
We showed that object-based representations, a context selection mechanism, factorization, and compositionality are useful ingredients for learning a physics simulator that generalizes across variable object count and different scene configurations with only spatially and temporally local computation. This generalization is possible because these ingredients transform the testing data distribution to be similar to the training data distribution.
The NPE makes few but strong assumptions about the nature of objects in a physical environment. These assumptions are inductive biases that not only give the NPE enough structure to help constrain it to model physical phenomena in terms of objects but also are general enough for the NPE to learn physical dynamics almost exclusively from observation.
We applied the NPE to simple two-dimensional worlds of bouncing balls ranging in complexity. We showed that NPE achieves low prediction error, extrapolates learned physical knowledge to previously unseen number of objects and world configurations, and can infer latent properties such as mass. We compared against several baselines designed to test the ingredients of the NPE framework and found superior performance when all these ingredients are combined in the NPE. Though we demonstrated the NPE in the balls environment with nonlinear dynamics and complex scene configurations, the state representation and NPE architecture we propose are quite general-purpose because they assume little about the specific dynamics of a scene.
This paper works toward emulating a general purpose physics engine under a framework where visual and physical aspects of a scene are disentangled. Next steps include linking the NPE with perceptual models that extract properties such as position and mass from visual input. Learning to simulate is unsupervised learning of the structure of the environment. When a simulator like the NPE is incorporated into an agent in the context of model-based planning and model-based reinforcement learning, it becomes a prior on the environment that guides learning and reasoning. By combining the expressiveness of physics engines and the adaptability of neural networks in a compositional architecture that supports generalization in fundamental aspects of physical reasoning, the Neural Physics Engine is an important step towards lifting an agent’s ability to think at a level of abstraction where the concept of physics is primitive.
ACKNOWLEDGMENTS
We thank Tejas Kulkarni for insightful discussions and guidance. We thank Ilker Yildirim, Erin Reynolds, Feras Saad, Andreas Stuhlmüller, Adam Lerer, Chelsea Finn, Jiajun Wu, and the anonymous reviewers for valuable feedback. We thank Liam Brummit, Kevin Kwok, and Guillermo Webster for help with matter-js. This work was supported MIT’s SuperUROP and UROP programs, and by the Center for Minds, Brains and Machines under NSF STC award CCF-1231216 and an ONR grant N00014-16-1-2007.
B QUANTITATIVE ANALYSIS

- summary
The paper proposes a differntiable Neural Physics Engine (NPE). The NPE consists of an encoder and a decoder function. The NPE takes as input the state of pairs of objects (within a neighbourhood of a focus object) at two previous time-steps in a scene. The encoder function summarizes the interaction of each pair of objects. The decoder then outputs the change in velocity of the focus object at the next time step. The NPE is evaluated on various environments containing bouncing balls.
- novelty
The differentiable NPE is a novel concept. However, concurrently Battaglia et al. (NIPS 2016) proposes a very similar model. Just as this work, Battaglia et al. (NIPS 2016) consider a model which consists of a encoder function (relation-centric) which encodes the interaction among a focus object and other objects in the scene and a decoder (relation-centric) function which considers the cumulative (encoded) effect of object interactions on the focus object and predicts effect of the interactions.  Aspects like only considering objects interactions within a neighbourhood (versus the complete object interaction graph in Battaglia et al.) based on euclideian distance  are novel to this work. However, the advantages (if any) of NPE versus the model of Battaglia et al. are not clear. Moreover, it is not clear how this neighbourhood thresholding scene would preform in case of n-ball systems, where gravitational forces of massive objects can be felt over large distances.
- citations
This work includes all relevant citations.
- clarity
The article is well written and easy to understand.
- experiments
Battaglia et al. evaluates on wider variety senerios compared to this work (e.g. n-bodies under gravitation, falling strings). Such experiments demonstrate the ability of the models to generalize. However, this work does include more in-depth experiments in case of bouncing balls compared to Battaglia et al. (e.g. mass estimation and varying world configurations with obstacles in the bouncing balls senerio).
Moreover, an extensive comparison to Fragkiadaki et al. (2015) (in the bouncing balls senerios) is missing. The authors (referring to answer to question 4) do point out to comaprable numbers in both works, but the experimental settings are different.  Comparison in a billiard table senerio like that Fragkiadaki et al. (2015) where a initial force is applied to a ball, would have been enlightening.
The authors only evaluate the error in velocity in the bouncing balls senerios. We understand that this model predicts only the velocity (refer to answer of question 2). Error analysis also with respect to ground truth ball position would be more enlightening. As small errors in velocity can quickly lead to entirely different scene configuration.
- conclusion / recommendation
The main issue with this work is the unclear novelty with respect to work of Battaglia et al. at NIPS'16. A quantitative and qualitative comparison with Battaglia et al. is lacking.  But the authors state that their work was developed independently.
Differentiable physics engines like NPE or that of Battaglia et al. (NIPS 2016) requires generation of an extensive amount of synthetic data to learn about the physics of a certain senerio. Moreover, extensive retraining is required to adapt to new sceneries (e.g. bouncing balls to n-body systems). Any practical advantage versus generating new code for a physics engine is not clear. Other "bottom-up" approaches like that of  Fragkiadaki et al. (2015) couple vision along with learning dynamics. However, they require very few input parameters (position, mass, current velocity, world configuration), as approximate parameter estimation can be done from the visual component.  Such approaches could be potentially more useful of a robot in "common-sense" everyday tasks (e.g. manipulation). Thus, overall potential applications of a differentiable physics engine like NPE is unclear.


DSD: DENSE-SPARSE-DENSE TRAINING FOR DEEP NEURAL NETWORKS
Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ’93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn’t change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.
1 INTRODUCTION
Deep neural networks (DNNs) have shown significant improvements in many application domains, ranging from computer vision (He et al. (2015)) to natural language processing (Luong et al. (2015)) and speech recognition (Amodei et al. (2015)). The abundance of powerful hardware makes it easier to train complicated DNN models with large capacities. The upside of complicated models is that they are very expressive and can capture the highly non-linear relationship between features and output. The downside of such large models is that they are prone to capturing the noise, rather than the intended pattern, in the training dataset. This noise does not generalize to new datasets, leading to over-fitting and a high variance.
∗Indicates equal contribution †Also at NVIDIA ‡Now at Google Brain. eriche@google.com
Dense
Pruning
Sparsity Constraint
Sparse
Increase Model Capacity
Re-Dense
Dense
end goto Sparse Phase for iterative DSD;
In contrast, simply reducing the model capacity would lead to the other extreme, causing a machine learning system to miss the relevant relationships between features and target outputs, leading to under-fitting and a high bias. Bias and variance are hard to optimize at the same time.
To solve this problem, we propose a dense-sparse-dense training flow (DSD), a novel training strategy that starts from a dense model from conventional training, then regularizes the model with sparsityconstrained optimization, and finally increases the model capacity by restoring and retraining the pruned weights. At testing time, the final model produced by DSD still has the same architecture and dimension as the original dense model, and DSD training doesn’t incur any inference overhead. We experimented DSD training on 7 mainstream CNN / RNN / LSTMs and found consistent performance gains over its comparable counterpart for image classification, image captioning and speech recognition.
2 DSD TRAINING FLOW
Our DSD training employs a three-step process: dense, sparse, re-dense. Each step is illustrated in Figure 1 and Algorithm 1. The progression of weight distribution is plotted in Figure 2.
Initial Dense Training: The first D step learns the connection weights and importance via normal network training on the dense network. Unlike conventional training, however, the goal of this D step is not only to learn the values of the weights; we are also learning which connections are important. We use a simple heuristic to quantify the importance of the weights using their absolute value.
Sparse Training: The S step prunes the low-weight connections and trains a sparse network. We applied the same sparsity to all the layers, thus there’s a single hyper parameter: the sparsity, the percentage of weights that are pruned to 0. For each layer W with N parameters, we sorted the parameters, picked the k-th largest one λ = Sk as the threshold where k = N ∗ (1− sparsity), and generated a binary mask to remove all the weights smaller than λ. Details are shown in Algorithm 1 .
We remove small weights because of the Taylor expansion. The loss function and its Taylor expansion are shown in Equation (1)(2). We want to minimize the increase in Loss when conducting a hard thresholding on the weights, so we need to minimize the first and second terms in Equation 2. Since we are zeroing out parameters, ∆Wi is actually Wi − 0 = Wi. At the local minimum where ∂Loss/∂Wi ≈ 0 and ∂
2Loss ∂W 2i > 0, only the second order term matters. Since second order gradient
∂2Loss/∂W 2i is expensive to calculate andWi has a power of 2, we use |Wi| as the metric of pruning. Smaller |Wi| means a smaller increase to the loss function.
Loss = f(x,W1,W2,W3...) (1)
∆Loss = ∂Loss
∂Wi ∆Wi +
1
2
∂2Loss
∂W 2i ∆Wi
2 + ... (2)
Retraining while enforcing the binary mask in each iteration, we converted a dense network into a sparse network that has a known sparsity support and can fully recover or even increase the original accuracy of initial dense model under the sparsity constraint. The sparsity is the same for all the layers and can be tuned using validation. We find a sparsity value between 25% and 50% generally works well in our experiments.
Final Dense Training: The final D step recovers the pruned connections, making the network dense again. These previously-pruned connections are initialized to zero and the entire network is retrained with 1/10 the original learning rate (since the sparse network is already at a good local minima). Hyper parameters like dropout ratios and weight decay remained unchanged. By restoring the pruned connections, the final D step increases the model capacity of the network and makes it possible to arrive at a better local minima compared with the sparse model from the S step.
To visualize the DSD training flow, we plotted the progression of the weight distribution in Figure 2. The figure is plotted using GoogLeNet’s inception_5b3x3 layer, and we found this progression of weight distribution very representative for VGGNet and ResNet as well. The original distribution of weight is centered on zero with tails dropping off quickly. Pruning is based on absolute value so after pruning the large center region is truncated away. The un-pruned network parameters adjust themselves during the retraining phase, so in (c), the boundary becomes soft and forms a bimodal distribution. In (d), at the beginning of the re-dense training step, all the pruned weights come back again and are reinitialized to zero. Finally, in (e), the pruned weights are retrained together with the un-pruned weights. In this step, we kept the same learning hyper-parameters (weight decay, learning rate, etc.) for pruned weights and un-pruned weights. Comparing Figure (d) and (e), the un-pruned weights’ distribution almost remained the same, while the pruned weights became distributed further around zero. The overall mean absolute value of the weight distribution is much smaller. This is a good phenomenon: choosing the smallest vector that solves the learning problem suppresses irrelevant components of the weight vector ( Moody et al. (1995)).
3 RELATED WORK
Dropout and DropConnect: DSD, Dropout (Srivastava et al. (2014)) and DropConnnect (Wan et al. (2013)) can all regularize neural networks and prevent over-fitting. The difference is that Dropout and DropConnect use a random sparsity pattern at each SGD iteration, while DSD training learns with a deterministic data driven sparsity pattern throughout sparse training. Our experiments on VGG16, GoogLeNet and NeuralTalk show that DSD training can work together with Dropout.
Distillation: Model distillation (Hinton et al. (2015)) is a method that can transfer the learned knowledge from a large model to a small model, which is more efficient for deployment. This is another method that allows for performance improvements in neural networks without architectural changes.
Model Compression: Both model compression (Han et al. (2016; 2015)) and DSD training use network pruning (LeCun et al. (1990); Hassibi et al. (1993)). The difference is that the focus of DSD training goes beyond maintaining the accuracy. DSD is able to further improve the accuracy by considerable margins. Another difference is that DSD training doesn’t require aggressive pruning. A modestly pruned network (50%-60% sparse) can work well. However, model compression requires aggressively pruning the network to achieve high compression rates.
Sparsity Regularization and Hard Thresholding: the truncation-based sparse network has been theoretically analyzed for learning a broad range of statistical models in high dimensions (Langford et al. (2009); Yuan & Zhang (2013); Wang et al. (2014)). A similar training strategy with iterative hard thresholding and connection restoration is proposed by Jin et al. (2016) during the same time period as, but independently from, DSD. Sparsity regularized optimization is heavily applied in Compressed Sensing (Candes & Romberg (2007)) to find optimal solutions to the inverse problems in highly under-determined systems based on the sparsity assumption.
4 EXPERIMENTS
We applied DSD training to different kinds of neural networks in different domains. We found that DSD training improved the accuracy for all these networks compared to the baseline networks that were not trained with DSD. The neural networks are chosen from CNN, RNN and LSTMs; the datasets covered image classification, speech recognition, and caption generation. For networks trained for ImageNet, we focus on GoogLeNet, VGG and ResNet, which are widely used in research and production. An overview of the networks, dataset and accuracy results are shown in Table 1. For the convolutional networks, we do not prune the first layer during the sparse phase, since it has only 3 channels and is very sensitive to pruning. The sparsity is the same for all the other layers, including convolutional and fully-connected layers. We do not change any other training hyper-parameters, and the initial learning rate at each stage is decayed the same as conventional training. The epochs are decided by when the loss converges. When the loss no longer decreases, we stop the training.
4.1 GOOGLENET
We experimented with the BVLC GoogLeNet (Szegedy et al. (2015)) model obtained from the Caffe Model Zoo (Jia (2013)). It has 13 million parameters and 57 convolutional layers. We pruned each layer (except the first) to 30% sparsity. Retraining the sparse network gave some improvement in accuracy due to regularization, as shown in Table 2. After the final dense training step, GoogLeNet’s error rates were reduced by 1.12% (Top-1) and 0.62% (Top-5) over the baseline.
We compared DSD v.s. conventional training for the same number of epochs by dropping the learning rate upon "convergence" and continuing to learn. The result is shown as LLR (lower the learning rate). The training epochs for LLR is equal to that of Sparse+re-Dense as a fair comparison. LLR can not achieve the same accuracy as DSD.
4.2 VGGNET
We explored DSD training on VGG-16 (Simonyan & Zisserman (2014)), which is widely used in detection, segmentation and transfer learning. The baseline model is obtained from the Caffe Model Zoo (Jia (2013)). Similar to GoogLeNet, each layer is pruned to 30% sparsity. DSD training greatly reduced the error by 4.31% (Top-1) and 2.65% (Top-5), detailed in Table 3. DSD also wins over the LLR result by a large margin.
4.3 RESNET
Deep Residual Networks (ResNets, He et al. (2015)) were the top performer in the 2015 ImageNet challenge. The baseline ResNet-18 and ResNet-50 models are provided by Facebook (2016). We prune to 30% sparsity uniformly, and a single DSD pass for these networks reduced top-1 error by 1.26% (ResNet-18) and 1.12% (ResNet-50), shown in Table 4. A second DSD iteration can further improve the accuracy. As a fair comparison, we continue train the original model by lowering the learning rate by another decade, but can’t reach the same accuracy as DSD, as shown in the LLR row.
1
4.4 NEURALTALK
We evaluated DSD training on RNN and LSTM beyond CNN. We applied DSD to NeuralTalk (Karpathy & Fei-Fei (2015)), an LSTM for generating image descriptions. It uses a CNN as an image feature extractor and an LSTM to generate captions. To verify DSD training on LSTMs, we fixed the CNN weights and only train the LSTM weights. The baseline NeuralTalk model we used is the flickr8k_cnn_lstm_v1.p downloaded from NeuralTalk Model Zoo.
In the pruning step, we pruned all layers except Ws, the word embedding lookup table, to 80% sparse. We used a higher sparsity than CNN’s experiments based on the validation set of flickr8k. We retrained the remaining sparse network using the same weight decay and batch size as the original paper. The learning rate is tuned based on the validation set, shown in Table 5. Retraining the sparse network improved the BLUE score by [1.2, 1.1, 0.9, 0.7]. After getting rid of the sparsity constraint and retraining the dense network, the final results of DSD further improved the BLEU score by [2.0, 2.1, 2.0, 1.7] over baseline.
The BLEU score is not the sole criteria measuring auto-caption system. We visualized the captions generated by DSD training in Figure 3. In the first image, the baseline model mistakes the girl with a boy and the girl’s hair with a rock wall; the sparse model can tell that it’s a girl; and the DSD model can further identify the swing. In the the second image, DSD training can more accurately tell the player is in a white uniform and trying to make a shot, rather than the baseline just saying he’s in a red uniform and playing with a ball. The performance of DSD training generalizes beyond these examples; more image caption results generated by DSD training are provided in the Appendix.
4.5 DEEPSPEECH
We explore DSD training on speech recognition tasks using both Deep Speech 1 (DS1) and Deep Speech 2 (DS2) networks (Hannun et al. (2014); Amodei et al. (2015)).
The DS1 model is a 5 layer network with 1 Bidirectional Recurrent layer, as described in Table 6. The training dataset used for this model is the Wall Street Journal (WSJ), which contains 81 hours of
speech. The validation set consists of 1 hour of speech. The test sets are from WSJ’92 and WSJ’93 and contain 1 hour of speech combined. The Word Error Rate (WER) reported on the test sets for the baseline models is different from Amodei et al. (2015) due to two factors. First, in DeepSpeech2, the models were trained using much larger data sets containing approximately 12,000 hours of multi-speaker speech data. Secondly, WER was evaluated with beam search and a language model in DeepSpeech2; here the network output is obtained using only max decoding to show improvement in the neural network accuracy, and filtering out the other parts.
The first dense phase was trained for 50 epochs. In the sparse phase, weights are pruned in the Fully Connected layers and the Bidirectional Recurrent layer only (they are the majority of the weights). Each layer is pruned to achieve the same 50% sparsity and trained for 50 epochs. In the final dense phase, the pruned weights are initialized to zero and trained for another 50 epochs. For a fair comparison of baseline, we used Nesterov SGD to train, reduce the learning rate with each re-training, and keep all other hyper parameters unchanged. The learning rate is picked using our validation set.
We first wanted to compare the DSD results with a baseline model trained for the same number of epochs. The first 3 rows of Table 7 shows the WER when the DSD model is trained for 50+50+50=150 epochs, and the 6th line shows the baseline model trained by 150 epochs (the Same #Epochs as DSD). DSD training improves WER by 0.13 (WSJ ’92) and 1.35 (WSJ ’93) given the same number of epochs as the conventional training.
Given a second DSD iteration, accuracy can be further improved. In the second DSD iteration, each layer is pruned away 25% of the weights. Similar to the first iteration, the sparse model and subsequent dense model are further retrained for 50 epochs. The learning rate is scaled down for each re-training step. The results are shown in Table 7. Compared with the fully trained and converged baseline, the second DSD iteration improves WER by 0.58 (WSJ ’92) and 1.96 (WSJ ’93), a relative improvement of 2.07% (WSJ ’92) and 5.84% (WSJ ’93). So, we can do more DSD iterations (DSDSD) to further improve the performance. Adding more DSD iterations has a diminishing return.
4.6 DEEPSPEECH 2
To show how DSD works on deeper networks, we evaluated DSD on the Deep Speech 2 (DS2) network, described in Table 8. This network has 7 Bidirectional Recurrent layers with approximately 67 million parameters, around 8 times larger than the DS1 model. A subset of the internal English training set is used. The training set is comprised of 2,100 hours of speech. The validation set is comprised of 3.46 hours of speech. The test sets are from WSJ’92 and WSJ’93, which contain 1 hour of speech combined.
Table 9 shows the results of the two iterations of DSD training. For the first sparse re-training, similar to DS1, 50% of the parameters from the Bidirectional Recurrent Layers and Fully Connected
Layers are pruned. The Baseline model is trained for 60 epochs to provide a fair comparison with DSD training. The baseline model shows no improvement after 40 epochs. With one iteration of DSD training, WER improves by 0.44 (WSJ ’92) and 0.56 (WSJ ’93) compared to the fully trained baseline.
Here we show again that DSD can be applied multiple times or iteratively for further performance gain. A second iteration of DSD training achieves better accuracy as shown in Table 9. For the second sparse iteration, 25% of parameters in the Fully Connected layer and Bidirectional Recurrent layers are pruned. Overall DSD training achieves relative improvement of 5.55% (WSJ ’92) and 7.44% (WSJ ’93) on the DS2 architecture. These results are in line with DSD experiments on the smaller DS1 network. We can conclude that DSD re-training continues to show improvement in accuracy with larger layers and deeper networks.
5 DISCUSSION
Dense-Sparse-Dense training changes the optimization process and improves the optimization performance with significant margins by nudging the network with pruning and re-densifying. We conjecture that the following aspects contribute to the efficacy of DSD training.
Escape Saddle Point: Based on previous studies, one of the most profound difficulties of optimizing deep networks is the proliferation of saddle points (Dauphin et al. (2014)). Advanced optimization methods have been proposed to overcome saddle points. For a similar purpose but with a different approach, the proposed DSD method overcomes the saddle points by pruning and re-densifying framework. Pruning the converged model perturbs the learning dynamics and allows the network to jump away from saddle points, which gives the network a chance to converge at a better local or global minimum. This idea is also similar to Simulated Annealing ( Hwang (1988)). While Simulated Annealing randomly jumps with decreasing probability on the search graph, DSD deterministically deviates from the converged solution achieved in the first dense training phase by removing the small weights and enforcing a sparsity support. Similar to Simulated Annealing, which can escape sub-optimal solutions multiple times in the entire optimization process, DSD can also be applied iteratively to achieve further performance gains, as shown in the Deep Speech results.
Significantly Better Minima: After escaping saddle point, DSD achieved better minima. We measured both the training loss and validation loss, DSD training decreased the loss and error on both the training and the validation sets on ImageNet. We have also validated the significance of the improvements compared with conventional fine-tuning by t-test, shown in the appendix.
Regularized and Sparse Training: The sparsity regularization in the sparse training step moves the optimization to a lower-dimensional space where the loss surface is smoother and tend to be more robust to noise. More numerical experiments verified that both sparse training and the final DSD reduce the variance and lead to lower error (shown in the appendix).
Robust re-initialization: Weight initialization plays a big role in deep learning (Mishkin & Matas (2015)). Conventional training has only one chance of initialization. DSD gives the optimization a second (or more) chance during the training process to re-initialize from a more robust sparse training solution. We re-densify the network from the sparse solution which can be seen as a zero initialization for pruned weights. Other initialization methods are also worth trying.
Break Symmetry: The permutation symmetry of the hidden units makes the weights symmetrical, thus prone to co-adaptation in training. In DSD, pruning the weights breaks the symmetry of the hidden units associated with the weights, and the weights are asymmetrical in the final dense phase.
6 CONCLUSION
We introduce DSD, a dense-sparse-dense training framework that regularizes neural networks by pruning and then restoring connections. Our method learns which connections are important during the initial dense solution. Then it regularizes the network by pruning the unimportant connections and retraining to a sparser and more robust solution with same or better accuracy. Finally, the pruned connections are restored and the entire network is retrained again. This increases the dimensionality of parameters, and thus model capacity, from the sparser model.
DSD training achieves superior optimization performance. We highlight our experiments using GoogLeNet, VGGNet, and ResNet on ImageNet; NeuralTalk on Flickr-8K; and DeepSpeech-1&2 on the WSJ dataset. This shows that the accuracy of CNNs, RNNs, and LSTMs can be significantly improved with DSD training. Our numerical results and empirical tests show the inadequacy of current training methods for which we have provided an effective solution.
A. APPENDIX: SIGNIFICANCE OF DSD IMPROVEMENTS
DSD training improves the baseline model performance by consecutively pruning and re-densifying the network weights. We conducted more intensive experiments to validate that the improvements are significant and not due to any randomness in the optimization. In order to evaluate the significance, we repeated the baseline training, DSD training (retraining on baseline) and conventional fine-tuning (retraining on the same baseline) multiple times. The statistical significance of DSD improvements are quantified on the Cifar-10 dataset using ResNet.
1. SIGNIFICANT IMPROVEMENTS ON CIFAR-10 USING RESNET-20
Cifar-10 is a smaller image recognition benchmark with 50,000 32x32 color images for training and 10,000 for testing. Training on Cifar-10 is fast enough that it is feasible to conduct intensive experiments within a reasonable time to evaluate DSD performance. The baseline models were trained with the standard 164 epochs and initial LR of 0.1 as recommended in the released code (Facebook, 2016). After 164 epochs, we obtained the model with a 8.26% top-1 testing error that is consistent with the Facebook result. Initialized from this baseline model, we repeated 16 times of re-training using DSD training and 16 times using conventional fine-tuning. The DSD used sparsity of 50% and 90 epochs (45 for sparse training and 45 for re-densing training). As a fair comparison, the conventional fine-tuning is also based on the same baseline model with the same hyper-parameters and settings (90 epochs, 45 LR of 0.001 and 45 LR of 0.0001).
Detailed results are listed below. On Cifar-10 and using ResNet-20 architecture, the DSD training on average achieved Top-1 testing error of 7.89%, which is a 0.37% absolute improvement (4.5% relative improvement) over the baseline model and relatively 1.1% better than the conventional fine-tuning. The experiment also shows that DSD training can reduce the variance of learning: the trained models after the sparse training and the final DSD training both have lower standard deviation of errors compared with their counterparts using conventional fine-tuning.
We used t-test (unpaired) to compare the top-1 testing error rate of the models trained using DSD and conventional methods. The results demonstrate the DSD training achieves significant improvements from both the baseline model (p<0.001) and conventional fine tuning (p<0.001).
Based on the results above, DSD significantly improves conventional baseline training and is also significantly better and more robust than conventional fine-tuning.
B. APPENDIX: MORE EXAMPLES OF DSD TRAINING IMPROVES THE CAPTIONS GENERATED BY NEURALTALK (IMAGES FROM FLICKR-8K TEST SET)
Baseline: a man in a red shirt and jeans is riding a bicycle down a street. Sparse: a man in a red shirt and a woman in a wheelchair. DSD: a man and a woman are riding on a street. Baseline: two girls in bathing suits are playing in the water. Sparse: two children are playing in the sand. DSD: two children are playing in the sand. Baseline: a group of people are standing in front of a building. Sparse: a group of people are standing in front of a building. DSD: a group of people are walking in a park.
Baseline: a dog runs through the grass. Sparse: a dog runs through the grass. DSD: a white and brown dog is running through the grass. Baseline: a group of football players in red uniforms. Sparse: a group of football players in a field. DSD: a group of football players in red and white uniforms. Baseline: a group of people sit on a bench in front of a building. Sparse: a group of people are standing in front of a building. DSD: a group of people are standing in a fountain. Baseline: a man in a black jacket and a black jacket is smiling. Sparse: a man and a woman are standing in front of a mountain. DSD: a man in a black jacket is standing next to a man in a black shirt.
Baseline:a young girl in a red dress is holding a camera. Sparse: a little girl in a pink dress is standing in front of a tree. DSD: a little girl in a red dress is holding a red and white flowers. Baseline: a man in a red jacket is standing in front of a white building. Sparse: a man in a black jacket is standing in front of a brick wall. DSD: a man in a black jacket is standing in front of a white building. Baseline: a man in a red shirt is standing on a rock. Sparse: a man in a red jacket is standing on a mountaintop. DSD: a man is standing on a rock overlooking the mountains. Baseline: a group of people are sitting in a subway station. Sparse: a man and a woman are sitting on a couch. DSD: a group of people are sitting at a table in a room.
Baseline: a soccer player in a red and white uniform is running on the field. Sparse: a soccer player in a red uniform is tackling another player in a white uniform. DSD: a soccer player in a red uniform kicks a soccer ball. Baseline: a young girl in a swimming pool. Sparse: a young boy in a swimming pool. DSD: a girl in a pink bathing suit jumps into a pool. Baseline: a soccer player in a red and white uniform is playing with a soccer ball. Sparse: two boys playing soccer. DSD: two boys playing soccer. Baseline: a girl in a white dress is standing on a sidewalk. Sparse: a girl in a pink shirt is standing in front of a white building. DSD: a girl in a pink dress is walking on a sidewalk.
Baseline: a boy is swimming in a pool. Sparse: a small black dog is jumping into a pool. DSD: a black and white dog is swimming in a pool.
A. Supplementary Material: More Examples of DSD Training Improves the Performance of NeuralTalk Auto-Caption System
1
Baseline: a snowboarder flies through the air. Sparse: a person is snowboarding down a snowy hill. DSD: a person on a snowboard is jumping over a snowy hill. Baseline: two young girls are posing for a picture. Sparse: a young girl with a blue shirt is blowing bubbles. DSD: a young boy and a woman smile for the camera. Baseline: a man in a red shirt is sitting in a subway station. Sparse: a woman in a blue shirt is standing in front of a store. DSD: a man in a black shirt is standing in front of a restaurant. Baseline: a surfer is riding a wave. Sparse: a man in a black wetsuit is surfing on a wave. DSD: a man in a black wetsuit is surfing a wave.
Baseline: a man in a red shirt is standing on top of a rock. Sparse: a man in a red shirt is standing on a cliff overlooking the mountains. DSD: a man is standing on a rock overlooking the mountains.
Baseline: a group of people sit on a bench. Sparse: a group of people are sitting on a bench. DSD: a group of children are sitting on a bench.
Baseline: a little boy is playing with a toy. Sparse: a little boy in a blue shirt is playing with bubbles. DSD: a baby in a blue shirt is playing with a toy.
Baseline: a brown dog is running through the grassy. Sparse: a brown dog is playing with a ball. DSD: a brown dog is playing with a ball.
Baseline: a boy in a red shirt is jumping on a trampoline. Sparse: a boy in a red shirt is jumping in the air. DSD: a boy in a red shirt is jumping off a swing.
Baseline: a man is standing on the edge of a cliff. Sparse: a man is standing on the shore of a lake. DSD: a man is standing on the shore of the ocean.
Baseline: two people are riding a boat on the beach. Sparse: two people are riding a wave on a beach. DSD: a man in a yellow kayak is riding a wave.
Baseline: a black and white dog is running on the beach. Sparse: a black and white dog running on the beach. DSD: a black dog is running on the beach.
Baseline: a man and a dog are playing with a ball. Sparse: a man and a woman are playing tug of war. DSD: a man and a woman are playing with a dog.
Baseline: a group of people are standing in a room. Sparse: a group of people gather together. DSD: a group of people are posing for a picture.
Baseline: a man in a red jacket is riding a bike through the woods. Sparse: a man in a red jacket is doing a jump on a snowboard. DSD: a person on a dirt bike jumps over a hill.
Baseline: a man in a red jacket and a helmet is standing in the snow. Sparse: a man in a red jacket and a helmet is standing in the snow. DSD: a man in a red jacket is standing in front of a snowy mountain.
2

Summary:
The paper proposes a model training strategy to achieve higher accuracy. The issue is train a too large model and you going to over-fit and your model will capture noise. Prune models or make it too small then it will miss important connections and under-fit. Thus, the proposed method involves various training steps: first they train a dense network, then prune it making it sparse then train a sparse network and finally they add connections back and train the model as dense again (DSD). The DSD method is generic method that can be used in CNN/RNN/LSTM. The reasons why models have better accuracy after DSD are: escape of saddle point, sparsity makes model more robust to noise and symmetry break allowing richer representations.
Pro:
The main point that this paper wants to show is that a model has the capacity to achieve higher accuracy, because it was shown that it is possible to compress a model without losing accuracy. And lossless compression means that there’s significant redundancy in the models that were trained using current training methods. This is an important observation that large models can get better accuracies as better training schemes are used.
Cons & Questions:
The issue is that the accuracy is slightly increased (2 or 3%) for most models. And the question is what is the price paid for this improvement? Resource and performance concerns arises because training a large model is computationally expensive (hours or even days using high performance GPUs).
Second question, can I keep adding Dense, Sparse and Dense training iterations to get higher and higher accuracy improvement? Are there limitations to this DSDSD… approach?