We introduce several techniques for sampling and visualizing the latent spaces of generative models. Replacing linear interpolation with spherical linear interpolation prevents diverging from a model’s prior distribution and produces sharper samples. J-Diagrams and MINE grids are introduced as visualizations of manifolds created by analogies and nearest neighbors. We demonstrate two new techniques for deriving attribute vectors: bias-corrected vectors with data replication and synthetic vectors with data augmentation. Binary classification using attribute vectors is presented as a technique supporting quantitative analysis of the latent space. Most techniques are intended to be independent of model type and examples are shown on both Variational Autoencoders and Generative Adversarial Networks.
1 INTRODUCTION
Generative models are a popular approach to unsupervised machine learning. Generative neural network models are trained to produce data samples that resemble the training set. Because the number of model parameters is significantly smaller than the training data, the models are forced to discover efficient data representations. These models are sampled from a set of latent variables in a high dimensional space, here called a latent space. Latent space can be sampled to generate observable data values. Learned latent representations often also allow semantic operations with vector space arithmetic (Figure 1).
Generative models are often applied to datasets of images. Two popular generative models for image data are the Variational Autoencoder (VAE, Kingma & Welling, 2014) and the Generative Adversarial Network (GAN, Goodfellow et al., 2014). VAEs use the framework of probabilistic graphical models with an objective of maximizing a lower bound on the likelihood of the data. GANs instead formalize
the training process as a competition between a generative network and a separate discriminative network. Though these two frameworks are very different, both construct high dimensional latent spaces that can be sampled to generate images resembling training set data. Moreover, these latent spaces are generally highly structured and can enable complex operations on the generated images by simple vector space arithmetic in the latent space (Larsen et al., 2016).
Generative models are beginning to find their way out of academia and into creative applications. In this paper we present techniques for improving the visual quality of generative models that are generally independent of the model itself. These include spherical linear interpolation, visualizing analogies with J-diagrams, and generating local manifolds with MINE grids. These techniques can be combined generate low dimensional embeddings of images close to the trained manifold. These can be used for visualization and creating realistic interpolations across latent space. Also by standardizing these operations independent of model type, the latent space of different generative models are more directly comparable with each other, exposing the strengths and weaknesses of various approaches.
Additionally, two new techniques for building latent space attribute vectors are introduced. On labeled datasets with correlated labels, data replication can be used to create bias-corrected vectors. Synthetic attributes vectors also can be derived via data augmentation on unlabeled data. Quantitative analysis of attribute vectors can be performed by using them as the basis for attribute binary classifiers.
2 SAMPLING TECHNIQUES
Generative models are often evaluated by examining samples from the latent space. Techniques frequently used are random sampling and linear interpolation. But often these can result in sampling the latent space from locations very far outside the manifold of probable locations.
Our work has followed two useful principles when sampling the latent space of a generative model. The first is to avoid sampling from locations that are highly unlikely given the prior of the model. This technique is very well established - including being used in the original VAE paper which adjusted sampling through the inverse CDF of the Gaussian to accommodate the Gaussian prior (Kingma & Welling, 2014). The second principle is to recognize that the dimensionality of the latent space is often artificially high and may contains dead zones that are not on the manifold learned during training. This has been demonstrated for VAE models (Makhzani et al., 2016) and implies that simply matching the model’s prior will not always be sufficient to yield samples that appear to have been drawn from the training set.
2.1 INTERPOLATION
Interpolation is used to traverse between two known locations in latent space. Research on generative models often uses interpolation as a way of demonstrating that a generative model has not simply memorized the training examples (eg: Radford et al., 2015, §6.1). In creative applications interpolations can be used to provide smooth transitions between two decoded images.
Frequently linear interpolation is used, which is easily understood and implemented. But this is often inappropriate as the latent spaces of most generative models are high dimensional (> 50 dimensions) with a Gaussian or uniform prior. In such a space, linear interpolation traverses locations that are extremely unlikely given the prior. As a concrete example, consider a 100 dimensional space with the Gaussian prior µ=0, σ=1. Here all random vectors will generally a length very close to 10 (standard deviation < 1). However, linearly interpolating between any two will usually result in a "tent-pole" effect as the magnitude of the vector decreases from roughly 10 to 7 at the midpoint, which is over 4 standard deviations away from the expected length.
Our proposed solution is to use spherical linear interpolation slerp instead of linear interpolation. We use a formula introduced by (Shoemake 85) in the context of great arc inbetweening for rotation animations:
This treats the interpolation as a great circle path on an n-dimensional hypersphere (with elevation changes). This technique has shown promising results on both VAE and GAN generative models and with both uniform and Gaussian priors (Figure 2).
2.2 ANALOGY
Analogy has been shown to capture regularities in continuous space models. In the latent space of some linguistic models “King – Man + Woman” results in a vector very close to “Queen” (Mikolov et al., 2013). This technique has also been used in the context of deep generative models to solve visual analogies (Reed et al., 2015).
Analogies are usually written in the form:
A : B :: C :?
Such a formation answers the question “What is the result of applying the transformation A:B to C?” In a vector space the solution generally proposed is to solve the analogy using vector math:
(B −A) = (?− C)
? = C +B −A
Note that an interesting property of this solution is an implied symmetry:
A : C :: B :?
Because the same terms can be rearranged:
(C −A) = (?−B)
? = B + C −A
Generative models that include an encoder for computing latent vectors given new samples allow for visual analogies. We have devised a visual representation for depicting analogies of visual generative networks called a “J-Diagram”. The J- Diagram uses interpolation across two dimensions two expose the manifold of the analogy. It also makes the symmetric nature of the analogy clear (Figure 3).
The J-Diagram also serves as a reference visualization across different model settings because it is deterministically generated from images which can be held constant. This makes it a useful tool for comparing results across epochs during training, after adjusting hyperparameters, or even across completely different model types (Figure 4).
2.3 MANIFOLD TRAVERSAL
Generative models can produce a latent space that is not tightly packed, and the dimensionality of the latent space is often set artificially high. As a result, the manifold of trained examples is can be a subset of the latent space after training, resulting in dead zones in the expected prior.
If the model includes an encoder, one simple way to stay on the manifold is by only using out of sample encodings (ie: any data not used in training) in the latent space. This is a useful diagnostic,
but is overly restrictive in a creative application context since it prevents the model from suggesting new and novel samples from the model. However, we can recover this ability by also including the results of operations on these encoded samples that stay close to the manifold, such as interpolation, extrapolation, and analogy generation.
Ideally, there would be a mechanism to discover this manifold within the latent space. In generative models with an encoder and ample out-of-sample data, we can instead precompute locations on the manifold with sufficient density, and later query for nearby points in the latent space from this known set. This offers a navigation mechanism based on hopping to nearest neighbors across a large database of encoded samples. When combined with interpolation, we call this visualization a Manifold Interpolated Neighbor Embedding (MINE). A MINE grid is useful to visualize local patches of the latent space (Figure 5).
3 ATTRIBUTE VECTORS
Many generative models result in a latent space that is highly structured, even on purely unsupervised datasets (Radford et al., 2015). When combined with labeled data, attribute vectors can be computed using simple arithmetic. For example, a vector can be computed which represents the smile attribute, which by shorthand we call a smile vector. Following (Larsen et al., 2016), the smile vector can be
computed by simply subtracting the mean vector for images without the smile attribute from the mean vector for images with the smile attribute. This smile vector can then be applied to in a positive or negative direction to manipulate this visual attribute on samples taken from latent space (Figure 6).
3.1 CORRELATED LABELS
The approach of building attribute vectors from means of labeled data has been noted to suffer from correlated labels (Larsen et al., 2016). While many correlations would be expected from ground truths (eg: heavy makeup and wearing lipstick) we discovered others that appear to be from sampling bias. For example, male and smiling attributes have unexpected negative correlations because women in the CelebA dataset are much more likely to be smiling than men. (Table 1).
In an online service we setup to automatically add and remove smiles from images1, we discovered this gender bias was visually evident in the results. Our solution was to use replication on the training data such that the dataset was balanced across attributes. This was effective because ultimately the vectors are simply summed together when computing the attribute vector (Figure 7).
1https://twitter.com/smilevector
This balancing technique can also be applied to attributes correlated due to ground truths. Decoupling attributes allows individual effects to be applied separately. As an example, the two attributes smiling and mouth open are highly correlated in the CelebA training set (Table 2). This is not surprising, as physically most people photographed smiling would also have their mouth open. However by forcing these attributes to be balanced, we can construct two decoupled attribute vectors. This allows for more flexibility in applying each attribute separately to varying degrees (Figure 8).
3.2 SYNTHETIC ATTRIBUTES
It has been noted that samples drawn from VAE based models is that they tend to be blurry (Goodfellow et al., 2014; Larsen et al., 2015). A possible solution to this would be to discover an attribute vector for “unblur”, and then apply this as a constant offset to latent vectors before decoding. CelebA includes a blur label for each image, and so a blur attribute vector was computed and then extrapolated in the negative direction. This was found to noticeably reduce blur, but also resulted in a number of unwanted artifacts such as increased image brightness.
We concluded this to be the result of human bias in labeling. Labelers appear more likely to label darker images as blurry, so this unblur vector was found to suffer from attribute correlation that also “lightened” the reconstruction. This bias could not be easily corrected because CelebA does not include a brightness label for rebalancing the data.
For the blurring attribute, an algorithmic solution is available. We take a large set of images from the training set and process them through a Gaussian blur filter (figure 9). Then we run both the original image set and the blurred image set through the encoder and subtract the means of each set to compute a new attribute vector for blur. We call this a synthetic attribute vector because this label is derived from algorithmic data augmentation of the training set. This technique removes the labeler bias, is straightforward to implement, and resulted in samples closely resembling the reconstructions with less noticeable blur (figure 10).
3.3 QUANTITATIVE ANALYSIS WITH CLASSICICATION
The results of applying attribute vectors visually to images are often quite striking. But it would be useful to be able to evaluate the relative efficacy of various attribute vectors to each other or across different models and hyperparamaters. Attribute vectors have potential to be widely applicable outside of images in domain specific latent spaces, and this provides additional challenges for quantifying their performance. For example, recent work has shown the ability to use latent space model as a continuous representation of molecules (Gómez 16); though it is straightforward to generate vectors for attributes such as solubility in these spaces, the evaluation of these operations to existing molecules would appear to require specific domain knowledge.
We have found that very effective classifiers can be built in latent space using the dot product of an encoded vector with a computed attribute vector (figure 11A, 11B). We have named this technique for building classifiers on latent spaces AtDot, and models trained on unsupervised data have been shown to produce strong results on CelebA across most attributes (table 3). Importantly, these binary
classifiers provide a quantitative basis to evaluate attribute vectors on a related surrogate task - their ability to be the basis for a simple linear classifier. This task is one of the most established paradigms in machine learning and provides a set of established tools such as ROC curves for offering new insights into the behavior of the attribute vectors across different hyperparameters or different models (figure 11C).
4 FUTURE WORK
Software to support most techniques presented in this paper is included in a python software library that can be used with various generative models2. We hope to continue to improve the library so that the techniques are applicable across a broad range of generative models. Attribute vector based classifiers also offer a promising way to evaluate the suitability of attribute vectors in domains outside of images.
We are investigating constructing a specially constructed prior on the latent space such that interpolations could be linear. This would simplify many of the latent space operations and might enable new types of operations.
Given sufficient test data, the extent to which an encoded dataset deviates from the expected prior should be quantifiable. Developing such a metric would be useful in understanding the structure of the different latent spaces including probability that random samples fall outside of the expected manifold of encoded data.
ACKNOWLEDGMENTS
I am thankful for the constructive feedback from readers including Ehud Ben-Reuven, Zachary Lipton, and Alex Champandard. I thank Victoria University of Wellington School of Design for supporting research on Creative Intelligence. I also thank the vibrant machine learning and creative coding communities on twitter for their support and encouragement.

This paper proposed a set of different things under the name of "sampling generative models", focusing on analyzing the learned latent space and synthesizing desirable output images with certain properties for GANs.  This paper does not have one single clear message or idea, but rather proposed a set of techniques that seem to produce visually good looking results.  While this paper has some interesting ideas, it also has a number of problems.
The spherical interpolation idea is interesting, but after a second thought this does not make much sense.  The proposed slerp interpolation equation (page 2) implicitly assumes that the two points q1 and q2 lie on the same sphere, in which case the parameter theta is the angle corresponding to the great arc connecting the two points on the sphere.  However, the latent space of a GAN, no matter trained with a uniform distribution or a Gaussian distribution, is not a distribution on a sphere, and many points have different distances to the origin.  The author's justification for this comes from the well known fact that in high dimensional space, even with a uniform distribution most points lie on a thin shell in the unit cube.  This is true because in high-dimensional space, the outer shell takes up most of the volume in space, and the inner part takes only a very small fraction of the space, in terms of volume.  This does not mean the density of data in the outer shell is greater than the inner part, though.  In a uniform distribution, the data density should be equal everywhere, a point on the outer shell is not more likely than a point in the inner part.  Under a Gaussian model, the data density is on the other hand higher in the center and much lower on the out side.  If we have a good model of data, then sampling the most likely points from the model should give us plausible looking samples.  In this sense, spherical interpolation should do no better than the normally used linear interpolation.  From the questions and answers it seems that the author does not recognize this distinction.  The results shown in this paper seem to indicate that spherical interpolation is better visually, but it is rather hard to make any concrete conclusions from three pairs of examples.  If this is really the case then there must be something else wrong about our understanding of the learned model.
Aside from these, the J-diagram and the nearest neighbor latent space traversal both seems to be good ways to explore the latent space of a learned model.  The attribute vector section on transforming images to new ones with desired attributes is also interesting, and it provides a few new ways to make the GAN latent space more interpretable.
Overall I feel most of the techniques proposed in this paper are nice visualization tools.  The contributions however, are mostly on the design of the visualizations, and not much on the technical and model side.  The spherical interpolation provides the only mathematical equation in the paper, yet the correctness of the technique is arguable.  For the visualization tools, there are also no quantitative evaluation, maybe these results are more art than science.


LEARNING LOCOMOTION SKILLS USING DEEPRL: DOES
The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gaitcycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.
The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gaitcycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.
1 INTRODUCTION
The introduction of deep learning models to reinforcement learning (RL) has enabled policies to operate directly on high-dimensional, low-level state features. As a result, deep reinforcement learning (DeepRL) has demonstrated impressive capabilities, such as developing control policies that can map from input image pixels to output joint torques (Lillicrap et al., 2015). However, the quality and robustness often falls short of what has been achieved with hand-crafted action abstractions, e.g., Coros et al. (2011); Geijtenbeek et al. (2013). While much is known about the learning of state representations, the choice of action parameterization is a design decision whose impact is not yet well understood.
Joint torques can be thought of as the most basic and generic representation for driving the movement of articulated figures, given that muscles and other actuation models eventually result in joint torques. However this ignores the intrinsic embodied nature of biological systems, particularly the synergy between control and biomechanics. Passive-dynamics, such as elasticity and damping from muscles and tendons, play an integral role in shaping motions: they provide mechanisms for energy storage, and mechanical impedance which generates instantaneous feedback without requiring any explicit computation. Loeb coins the term preflexes (Loeb, 1995) to describe these effects, and their impact on motion control has been described as providing intelligence by mechanics (Blickhan et al., 2007). This can also be thought of as a kind of partitioning of the computations between the control and physical system.
In this paper we explore the impact of four different actuation models on learning to control dynamic articulated figure locomotion: (1) torques (Tor); (2) activations for musculotendon units (MTU); (3) target joint angles for proportional-derivative controllers (PD); and (4) target joint velocities (Vel). Because Deep RL methods are capable of learning control policies for all these models, it now becomes possible to directly assess how the choice of actuation model affects the learning difficulty. We also assess the learned policies with respect to robustness, motion quality, and policy query rates. We show that action spaces which incorporate local feedback can significantly improve learning speed and performance, while still preserving the generality afforded by torque-level control. Such parameterizations also allow for more complex body structures and subjective improvements in motion quality.
Our specific contributions are: (1) We introduce a DeepRL framework for motion imitation tasks; (2) We evaluate the impact of four different actuation models on learned control policies according to four criteria; and (3) We propose an optimization approach that combines policy learning and actuator optimization, allowing neural networks to effective control complex muscle models.
2 BACKGROUND
Our task will be structured as a standard reinforcement problem where an agent interacts with its environment according to a policy in order to maximize a reward signal. The policy π(s, a) = p(a|s) represents the conditional probability density function of selecting action a ∈ A in state s ∈ S. At each control step t, the agent observes a state st and samples an action at from π. The environment in turn responds with a scalar reward rt, and a new state s′t = st+1 sampled from its dynamics p(s′|s, a). For a parameterized policy πθ(s, a), the goal of the agent is learn the parameters θ which maximizes the expected cumulative reward
J(πθ) = E [ T∑ t=0 γtrt ∣∣∣∣∣πθ ]
with γ ∈ [0, 1] as the discount factor, and T as the horizon. The gradient of the expected reward OθJ(πθ) can be determined according to the policy gradient theorem (Sutton et al., 2001), which provides a direction of improvement to adjust the policy parameters θ.
OθJ(πθ) = ∫ S dθ(s) ∫ A Oθlog(πθ(s, a))A(s, a)da ds
where dθ(s) = ∫ S ∑T t=0 γ
tp0(s0)p(s0 → s|t, πθ)ds0 is the discounted state distribution, where p0(s) represents the initial state distribution, and p(s0 → s|t, πθ) models the likelihood of reaching state s by starting at s0 and following the policy πθ(s, a) for t steps (Silver et al., 2014). A(s, a) represents a generalized advantage function. The choice of advantage function gives rise to a family of policy gradient algorithms, but in this work, we will focus on the one-step temporal difference advantage function (Schulman et al., 2015)
A(st, at) = rt + γV (s ′ t)− V (st) where V (s) = E [∑T
t=0 γ trt ∣∣∣s0 = s, πθ] is the state-value function, and can be defined recursively via the Bellman equation
V (st) = E rt,s′t
[rt + γV (s ′ t)|st, πθ]
A parameterized value function Vφ(s), with parameters φ, can be learned iteratively in a manner similar to Q-Learning by minimizing the Bellman loss,
L(φ) = E st,rt,s′t
[ 1
2 (yt − Vφ(st))2
] , yt = rt + γVφ(s ′ t)
πθ and Vφ can be trained in tandem using an actor-critic framework (Konda & Tsitsiklis, 2000).
In this work, each policy will be represented as a gaussian distribution with a parameterized mean µθ(s) and fixed covariance matrix Σ = diag{σ2i }, where σi is manually specified for each action parameter. Actions can be sampled from the distribution by applying gaussian noise to the mean action at = µθ(st) +N (0,Σ) The corresponding policy gradient will assume the form
OθJ(πθ) = ∫ S dθ(s) ∫ A Oθµθ(s)Σ −1 (a− µθ(s))A(s, a)da ds
which can be interpreted as shifting the mean of the action distribution towards actions that lead to higher than expected rewards, while moving away from actions that lead to lower than expected rewards.
3 TASK REPRESENTATION
3.1 REFERENCE MOTION
In our task, the goal of a policy is to imitate a given reference motion {q∗t } which consists of a sequence of kinematic poses q∗t in reduced coordinates. The reference velocity q̇∗t at a given time t is approximated by finite-difference q̇∗t ≈ q∗t+4t−q ∗ t
4t . Reference motions are generated via either using a recorded simulation result from a preexisting controller (“Sim”), or via manually-authored keyframes. Since hand-crafted reference motions may not be physically realizable, the goal is to closely reproduce a motion while satisfying physical constraints.
3.2 STATES
To define the state of the agent, a feature transformation Φ(q, q̇) is used to extract a set of features from the reduced-coordinate pose q and velocity q̇. The features consist of the height of the root (pelvis) from the ground, the position of each link with respect to the root, and the center of mass velocity of each link. When training a policy to imitate a cyclic reference motion {q∗t }, knowledge of the motion phase can help simplify learning. Therefore, we augment the state features with a set of target features Φ(q∗t , q̇t ∗), resulting in a combined state represented by st = (Φ(qt, q̇t),Φ(q∗t , q̇t ∗)). Similar results can also be achieved by providing a single motion phase variable as a state feature, as we show in Figure 15 (supplemental material).
3.3 ACTIONS
We train separate policies for each of the four actuation models, as described below. Each actuation model also has related actuation parameters, such as feedback gains for PD-controllers and musculotendon properties for MTUs. These parameters can be manually specified, as we do for the PD and Vel models, or they can be optimized for the task at hand, as for the MTU models. Table 1 provides a list of actuator parameters for each actuation model.
Target Joint Angles (PD): Each action represents a set of target angles q̂, where q̂i specifies the target angles for joint i. q̂ is applied to PD-controllers which compute torques according to τ i = kip(q̂ i − qi) + kid(ˆ̇qi − q̇i), where ˆ̇qi = 0, and kip and kid are manually-specified gains.
Target Joint Velocities (Vel): Each action specifies a set of target velocities ˆ̇q which are used to compute torques according to τ i = kid(ˆ̇q
i − q̇i), where the gains kid are specified to be the same as those used for target angles.
Torques (Tor): Each action directly specifies torques for every joint, and constant torques are applied for the duration of a control step. Due to torque limits, actions are bounded by manually specified limits for each joint. Unlike the other actuation models, the torque model does not require additional actuator parameters, and can thus be regarded as requiring the least amount of domain knowledge. Torque limits are excluded from the actuator parameter set as they are common for all parameterizations.
Muscle Activations (MTU): Each action specifies activations for a set of musculotendon units (MTU). Detailed modeling and implementation information are available in Wang et al. (2012). Each MTU is modeled as a contractile element (CE) attached to a serial elastic element (SE) and parallel elastic element (PE). The force exerted by the MTU can be calculated according to FMTU = FSE = FCE + FPE . Both FSE and FPE are modeled as passive springs, while FCE is actively controlled according to FCE = aMTUF0fl(lCE)fv(vCE), with aMTU being the muscle activation, F0 the maximum isometric force, lCE and vCE being the length and velocity of the contractile element. The functions fl(lCE) and fv(vCE) represent the force-length and force-velocity relationships, modeling the variations in the maximum force that can be exerted by a muscle as a function of its length and contraction velocity. Analytic forms are available in Geyer et al. (2003). Activations are bounded between [0, 1]. The length of each contractile element lCE are included as state features. To simplify control and reduce the number of internal state parameters per MTU, the policies directly control muscle activations instead of indirectly through excitations (Wang et al., 2012).
3.4 REWARD
The reward function consists of a weighted sum of terms that encourage the policy to track a reference motion.
r = wposerpose + wvelrvel + wendrend + wrootrroot + wcomrcom
wpose = 0.5, wvel = 0.05, wend = 0.15, wroot = 0.1, wcom = 0.2
Details of each term are available in the supplemental material. rpose penalizes deviation of the character pose from the reference pose, and rvel penalizes deviation of the joint velocities. rend and rroot accounts for the position error of the end-effectors and root. rcom penalizes deviations in the center of mass velocity from that of the reference motion.
3.5 INITIAL STATE DISTRIBUTION
We design the initial state distribution, p0(s), to sample states uniformly along the reference trajectory. At the start of each episode, q∗ and q̇∗ are sampled from the reference trajectory, and used to initialize the pose and velocity of the agent. This helps guide the agent to explore states near the target trajectory.
4 ACTOR-CRITIC LEARNING ALGORITHM
Instead of directly using the temporal difference advantage function, we adapt a positive temporal difference (PTD) update as proposed by Van Hasselt (2012).
A(s, a) = I [δ > 0] =
{ 1, δ > 0
0, otherwise
δ = r + γV (s′)− V (s) Unlike more conventional policy gradient methods, PTD is less sensitive to the scale of the advantage function and avoids instabilities that can result from negative TD updates. For a Gaussian policy, a negative TD update moves the mean of the distribution away from an observed action, effectively shifting the mean towards an unknown action that may be no better than the current mean action (Van Hasselt, 2012). In expectation, these updates converges to the true policy gradient, but for stochastic estimates of the policy gradient, these updates can cause the agent to adopt undesirable behaviours which affect subsequent experiences collected by the agent. Furthermore, we incorporate experience replay, which has been demonstrated to improve stability when training neural network policies with Q-learning in discrete action spaces. Experience replay often requires off-policy methods, such as importance weighting, to account for differences between the policy being trained and the behavior policy used to generate experiences (WawrzyńSki & Tanwani, 2013). However, we have not found importance weighting to be beneficial for PTD.
Stochastic policies are used during training for exploration, while deterministic policy are deployed for evaluation at runtime. The choice between a stochastic and deterministic policy can be specified by the addition of a binary indicator variable λ ∈ [0, 1]
at = µθ(st) + λN (0,Σ)
where λ = 1 corresponds to a stochastic policy with exploration noise, and λ = 0 corresponds to a deterministic policy that always selects the mean of the distribution. Noise from a stochastic policy will result in a state distribution that differs from that of the deterministic policy at runtime. To imitate this discrepancy, we incorporate -greedy exploration in addition to the original Gaussian exploration. During training, λ is determined by a Bernoulli random variable λ ∼ Ber( ), where λ = 1 with probability ∈ [0, 1]. The exploration rate is annealed linearly from 1 to 0.2 over 500k iterations, which slowly adjusts the state distribution encountered during training to better resemble the distribution at runtime. Since the policy gradient is defined for stochastic policies, only tuples recorded with exploration noise (i.e. λ = 1) can be used to update the actor, while the critic can be updated using all tuples.
Training proceeds episodically, where the initial state of each episode is sampled from p0(s), and the episode duration is drawn from an exponential distribution with a mean of 2s. To discourage falling, an episode will also terminate if any part of the character’s trunk makes contact with the ground for an extended period of time, leaving the agent with zero reward for all subsequent steps. Algorithm 1 in the supplemental material summarizes the complete learning process.
MTU Actuator Optimization: Actuation models such as MTUs are defined by further parameters whose values impact performance (Geijtenbeek et al., 2013). Geyer et al. (2003) uses existing anatomical estimates for humans to determine MTU parameters, but such data is not be available for more arbitrary creatures. Alternatively, Geijtenbeek et al. (2013) uses covariance matrix adaptation (CMA), a derivative-free evolutionary search strategy, to simultaneously optimize MTU and policy parameters. This approach is limited to policies with reasonably low dimensional parameter spaces, and is thus ill-suited for neural network models with hundreds of thousands of parameters. To avoid manual-tuning of actuator parameters, we propose a heuristic approach that alternates between policy learning and actuator optimization, as detailed in the supplemental material.
5 RESULTS
The motions are best seen in the supplemental video https://youtu.be/L3vDo3nLI98. We evaluate the action parameterizations by training policies for a simulated 2D biped, dog, and raptor as shown in Figure 1. Depending on the agent and the actuation model, our systems have 58–214 state dimensions, 6–44 action dimensions, and 0–282 actuator parameters, as summarized in Table 3 (supplemental materials). The MTU models have at least double the number of action parameters because they come in antagonistic pairs. As well, additional MTUs are used for the legs to more accurately reflect bipedal biomechanics. This includes MTUs that span multiple joints.
Each policy is represented by a three layer neural network, as illustrated in Figure 8 (supplemental material) with 512 and 256 fully-connected units, followed by a linear output layer where the number of output units vary according to the number of action parameters for each character and actuation model. ReLU activation functions are used for both hidden layers. Each network has approximately 200k parameters. The value function is represented by a similar network, except having a single linear output unit. The policies are queried at 60Hz for a control step of about 0.0167s. Each network is randomly initialized and trained for about 1 million iterations, requiring 32 million tuples, the equivalent of approximately 6 days of simulated time. Each policy requires about 10 hours for the biped, and 20 hours for the raptor and dog on an 8-core Intel Xeon E5-2687W.
Only the actuator parameters for MTUs are optimized with Algorithm 2, since the parameters for the other actuation models are few and reasonably intuitive to determine. The initial actuator parameters ψ0 are manually specified, while the initial policy parameters θ0 are randomly initialized. Each pass optimizes ψ using CMA for 250 generations with 16 samples per generation, and θ is trained for 250k iterations. Parameters are initialized with values from the previous pass. The expected value of each CMA sample of ψ is estimated using the average cumulative reward over 16 rollouts with a duration of 10s each. Separate MTU parameters are optimized for each character and motion. Each set of parameters is optimized for 6 passes following Algorithm 2, requiring approximately 50 hours. Figure 5 illustrates the performance improvement per pass. Figure 6 compares the performance of MTUs before and after optimization. For most examples, the optimized actuator parameters significantly improve learning speed and final performance. For the sake of comparison, after a set of actuator parameters has been optimized, a new policy is retrained with the new actuator parameters and its performance compared to the other actuation models.
Policy Performance and Learning Speed: Figure 2 shows learning curves for the policies and the performance of the final policies are summarized in Table 4. Performance is evaluated using the normalized cumulative reward (NCR), calculated from the average cumulative reward over 32 episodes with lengths of 10s, and normalized by the maximum and minimum cumulative reward possible for each episode. No discounting is applied when calculating the NCR. The initial state of each episode is sampled from the reference motion according to p(s0). To compare learning speeds, we use the normalized area under each learning curve (AUC) as a proxy for the learning speed of a particular actuation model, where 0 represents the worst possible performance and no progress during training, and 1 represents the best possible performance without requiring training.
PD performs well across all examples, achieving comparable-to-the-best performance for all motions. PD also learns faster than the other parameterizations for 5 of the 7 motions. The final performance of Tor is among the poorest for all the motions. Differences in performance appear more pronounced as characters become more complex. For the simple 7-link biped, most parameterizations achieve similar performance. However, for the more complex dog and raptor, the performance of Tor policies deteriorate with respect to other policies such as PD and Vel. MTU policies often exhibited the slowest learning speed, which may be a consequence of the higher dimensional action spaces, i.e., requiring antagonistic muscle pairs, and complex muscle dynamics. Nonetheless, once optimized, the MTU policies produce more natural motions and responsive behaviors as compared to other parameterizations. We note that the naturalness of motions is not well captured by the reward, since it primarily gauges similarity to the reference motion, which may not be representative of natural responses when perturbed from the nominal trajectory. A sensitivity analysis of the policies’ performance to variations in network architecture and hyperparameters are available in the supplemental material.
Policy Robustness: To evaluate robustness, we recorded the NCR achieved by each policy when subjected to external perturbations. The perturbations assume the form of random forces applied
to the trunk of the characters. Figure 3 illustrates the performance of the policies when subjected to perturbations of different magnitudes. The magnitude of the forces are constant, but direction varies randomly. Each force is applied for 0.1 to 0.4s, with 1 to 4s between each perturbation. Performance is estimated using the average over 128 episodes of length 20s each. For the biped walk, the Tor policy is significantly less robust than those for the other types of actions, while the MTU policy is the least robust for the raptor run. Overall, the PD policies are among the most robust for all the motions. In addition to external forces, we also evaluate robustness over randomly generated terrain consisting of bumps with varying heights and slopes with varying steepness. We evaluate the performance on irregular terrain (Figure 12, supplemental material). There are few discernible patterns for this test. The Vel and MTU policies are significantly worse than the Tor and PD policies for the dog bound on the bumpy terrain. The unnatural jittery behavior of the dog Tor policy proves to be surprisingly robust for this scenario. We suspect that the behavior prevents the trunk from contacting the ground for extended periods for time, and thereby escaping our system’s fall detection.
Query Rate: Figure 4 compares the performance of different parameterizations for different policy query rates. Separate policies are trained with queries of 15Hz, 30Hz, 60Hz, and 120Hz. Actuation models that incorporate low-level feedback such as PD and Vel, appear to cope more effectively to lower query rates, while the Tor degrades more rapidly at lower query rates. It is not yet obvious to us why MTU policies appear to perform better at lower query rates and worse at higher rates. Lastly, Figure 14 shows the policy outputs as a function of time for the four actuation models, for a particular joint, as well as showing the resulting joint torque. Interestingly, the MTU action is visibly smoother than the other actions and results in joint torques profiles that are smoother than those seen for PD and Vel.
6 RELATED WORK
DeepRL has driven impressive recent advances in learning motion control, i.e., solving for continuous-action control problems using reinforcement learning. All four of the actions types that we explore have seen previous use in the machine learning literature. WawrzyńSki & Tanwani (2013) use an actor-critic approach with experience replay to learn skills for an octopus arm (actuated by a simple muscle model) and a planar half cheetah (actuated by joint-based PD-controllers).
Recent work on deterministic policy gradients (Lillicrap et al., 2015) and on RL benchmarks, e.g., OpenAI Gym, generally use joint torques as the action space, as do the test suites in recent work (Schulman et al., 2015) on using generalized advantage estimation. Other recent work uses: the PR2 effort control interface as a proxy for torque control (Levine et al., 2015); joint velocities (Gu et al., 2016); velocities under an implicit control policy (Mordatch et al., 2015); or provide abstract actions (Hausknecht & Stone, 2015). Our learning procedures are based on prior work using actorcritic approaches with positive temporal difference updates (Van Hasselt, 2012).
Work in biomechanics has long recognized the embodied nature of the control problem and the view that musculotendon systems provide “preflexes” (Loeb, 1995) that effectively provide a form intelligence by mechanics (Blickhan et al., 2007), as well as allowing for energy storage. The control strategies for physics-based character simulations in computer animation also use all the forms of actuation that we evaluate in this paper. Representative examples include quadratic programs that solve for joint torques (de Lasa et al., 2010), joint velocities for skilled bicycle stunts (Tan et al., 2014), muscle models for locomotion (Wang et al., 2012; Geijtenbeek et al., 2013), mixed use of feed-forward torques and joint target angles (Coros et al., 2011), and joint target angles computed by learned linear (time-indexed) feedback strategies (Liu et al., 2016). Lastly, control methods in robotics use a mix of actuation types, including direct-drive torques (or their virtualized equivalents), series elastic actuators, PD control, and velocity control. These methods often rely heavily on modelbased solutions and thus we do not describe these in further detail here.
7 CONCLUSIONS
Our experiments suggest that action parameterizations that include basic local feedback, such as PD target angles, MTU activations, or target velocities, can improve policy performance and learning speed across different motions and character morphologies. Such models more accurately reflect the embodied nature of control in biomechanical systems, and the role of mechanical components in shaping the overall dynamics of motions and their control. The difference between low-level and high-level action parameterizations grow with the complexity of the characters, with high-level parameterizations scaling more gracefully to complex characters. As a caveat, there may well be tasks, such as impedance control, where lower-level action parameterizations such as Tor may prove advantageous. We believe that no single action parameterization will be the best for all problems. However, since objectives for motion control problems are often naturally expressed in terms of kinematic properties, higher-level actions such as target joint angles and velocities may be effective for a wide variety of motion control problems. We hope that our work will help open discussions around the choice of action parameterizations.
Our results have only been demonstrated on planar articulated figure simulations; the extension to 3D currently remains as future work. Furthermore, our current torque limits are still large as compared to what might be physically realizable. Tuning actuator parameters for complex actuation models such as MTUs remains challenging. Though our actuator optimization technique is able to improve performance as compared to manual tuning, the resulting parameters may still not be optimal for the desired task. Therefore, our comparisons of MTUs to other action parameterizations may not be reflective of the full potential of MTUs with more optimal actuator parameters. Furthermore, our actuator optimization currently tunes parameters for a specific motion, rather than a larger suite of motions, as might be expected in nature.
Since the reward terms are mainly calculated according to joint positions and velocities, it may seem that it is inherently biased in favour of PD and Vel. However, the real challenges for the control policies lie elsewhere, such as learning to compensate for gravity and ground-reaction forces, and learning foot-placement strategies that are needed to maintain balance for the locomotion gaits. The reference pose terms provide little information on how to achieve these hidden aspects of motion control that will ultimately determine the success of the locomotion policy. While we have yet to provide a concrete answer for the generalization of our results to different reward functions, we believe that the choice of action parameterization is a design decision that deserves greater attention regardless of the choice of reward function.
Finally, it is reasonable to expect that evolutionary processes would result in the effective co-design of actuation mechanics and control capabilities. Developing optimization and learning algorithms to allow for this kind of co-design is a fascinating possibility for future work.

Paper studies deep reinforcement learning paradigm for controlling high dimensional characters. Experiments compare the effect different control parameterizations (torques, muscle-activations, PD control with target joint positions and target joint velocities) have on the performance of reinforcement learning and optimized control policies. Evaluated are different planer gate cycle trajectories. It is illustrated that more abstract parameterizations are in fact better and result in more robust and higher quality policies.
> Significance & Originality:
The explored parameterizations are relatively standard in humanoid control. The real novelty is systematic evaluation of the various parameterizations. I think this type of study is important and insightful. However, the findings are very specific to the problem and specific tested architecture. Its not clear that findings will transferable to other networks on other control problems/domains. As such for the ICLR community, this may have limited breadth and perhaps would have broader appeal in robotics / graphics community.
> Clarity:
The paper is well written and is pretty easy to understand for someone who has some background with constrained multi-body simulation and control.
> Experiments:
Experimental validation is lacking somewhat in my opinion. Given that this is a fundamentally experimental paper, I would have liked to see more analysis of sensitivity to various parameters and analysis of variance of performance when policy is optimized multiple times.


Humans learn a predictive model of the world and use this model to reason about future events and the consequences of actions. In contrast to most machine predictors, we exhibit an impressive ability to generalize to unseen scenarios and reason intelligently in these settings. One important aspect of this ability is physical intuition (Lake et al., 2016). In this work, we explore the potential of unsupervised learning to find features that promote better generalization to settings outside the supervised training distribution. Our task is predicting the stability of towers of square blocks. We demonstrate that an unsupervised model, trained to predict future frames of a video sequence of stable and unstable block configurations, can yield features that support extrapolating stability prediction to blocks configurations outside the training set distribution.
1 INTRODUCTION
Humans learn a tremendous amount of knowledge about the world with almost no supervision and can construct a predictive model of the world. We use this model of the world to interact with our environment. As also argued by Lake et al. (2016) one of the core ingredients of human intelligence is intuitive physics. Children can learn and predict some of the common physical behaviors of our world just by observing and interacting without any direct supervision. And they form a sophisticated predictive model of the physical environment and expect the world to behave based on their mental model and have a reasonable expectation about unseen situations Téglás et al. (2011).
Despite impressive progress in the last few years in the training of the supervised models, we have not yet quite been able to achieve similar results in unsupervised learning, and it remains one of the challenging research areas in the field. The full potential of the application of unsupervised learning is yet to be realized.
In this work, we leverage unsupervised learning to train a predictive model over sequences. We use the imagined and predicted future sequence data to help a physical environment prediction model generalize better to unseen settings.
More specifically we focus on the task of predicting if a tower of square bricks will fall or not, as introduced by Lerer et al. (2016). They showed that a deep convolution neural network could predict the fall of the towers with super-human accuracy. But despite the strengths of convolution neural networks, Zhang et al. (2016) shows how deep neural networks have a hard time generalizing to novel situations in the same way as humans or simulation-based models can do. In this work, we show that deep neural networks are capable of generalizing to novel situations through a form of unsupervised learning. The core idea is to observe the world without any supervision and build a future predictive model of it, and in a later stage leverage and utilize the imagined future to train a better fall prediction model.
2 RELATED WORK
In the beginning, unsupervised learning and generative models emerged as pre-training method Hinton & Salakhutdinov (2006); Hinton et al. (2006); Bengio et al. (2007) to help other tasks such as
supervised learning. But since Krizhevsky et al. (2012) many other regularization Srivastava et al. (2014), weight initialization Glorot & Bengio (2010) and normalization Ioffe & Szegedy (2015) techniques and architecture designs He et al. (2015) has been introduced that diminish the effect of pre-training. Although pre-training still could be useful in data scarce domains they are many other ways and applications that unsupervised learning are still very interesting models and it is a very active area of research. Just to name a few applications are semi-supervised learning Kingma et al. (2014); Salimans et al. (2016); Dumoulin et al. (2016) super resolution Sønderby et al. (2016).
Video generation is one active area of research with many applications, and many of the recent works have been using some of the states of the art neural networks for video generation. Srivastava et al. (2015) uses LSTM recurrent neural networks to train an unsupervised future predictive model for video generation. And here we use a very similar architecture as described in Section 4.1. Mathieu et al. (2015) combines the common mean-squared-error objective function with an adversarial training cost in order to generate sharper samples. Lotter et al. (2016) introduce another form of unsupervised video prediction training scheme that manages to predict future events such as the direction of the turn of a car which could have potential use in training of the self-driving cars.
Model-based reinforcement learning (RL) is an active research area that holds the promise of making the RL agents less data hungry. Learning agents could explore, learn in an unsupervised way about their world, and learn even more by dreaming about future states. We believe that action-condition video prediction models are an important ingredient for this task. Fragkiadaki et al. (2015) learn the dynamics of billiards balls by supervised training of a neural net. Action-conditioned video prediction models have been applied to Atari playing agent Oh et al. (2015) as well as robotics (Finn et al., 2016; Finn & Levine, 2016).
3 DATASET
Recent datasets for predicting the stability of block configurations (Lerer et al., 2016; Zhang et al., 2016) only provide binary labels of stability, and exclude the video simulation of the block configuration. We, therefore, construct a new dataset, with a similar setup as Lerer et al. (2016); Zhang et al. (2016), that includes this video sequence. We use a Javascript based physics engine1 to generate the data.
We construct towers made of 3 − 5 square blocks. To sample a random tower configuration, we uniformly shift each block in its x and y position such that it touches the block below. Because taller towers are more unstable, this shift is smaller when we add more blocks. To simplify our learning setting, we balance the number of stable and unstable block configurations. For each tower height, we create 8000, 1000 and 3000 video clips for the training, validation, and test set, respectively. The video clips are sub-sampled in time to include more noticeable changes in the blocks configurations. We decided to keep 39 number of frames which with our sub-sampling rate was enough time for unstable towers to collapse. Each video frame is an RGB image of size 64x64. In addition to binary stability label, we include the number of blocks that fell down.
4 ARCHITECTURE
The core idea of this paper is to use future state predictions of a generative video model to enhance the performance of a supervised prediction model. Our architecture consists of two separate modules:
Frame predictor A generative model to predict future frames of a video sequence. This model is trained to either generate the last frame or the complete sequence of frames.
Stability predictor In the original task, stability is predicted from a static image of a block configuration. We explore whether, in addition to initial configuration, the last frame prediction of our unsupervised model improves the performance of the stability prediction.
In the following sections, we explore several different architectures for both modules.
1https://chandlerprall.github.io/Physijs/
4.1 FUTURE FRAME PREDICTION
We consider two different model architectures for this task. The first one, named ConvDeconv, only takes the first frame as input and predicts the last frame of the video sequence. The architecture consist of a block of convolution and max-pooling layers. To compensate for the dimensionality reduction of the max-pooling layers, we have a fully-connected layer following the last max-pooling layer. And finally a subsequent block of deconvolution layers with the output size same as the model input size. All activation functions are ReLU(Nair & Hinton, 2010). See Table 1 for more details of the architecture. The objective function is the mean squared error between the generated last frame and the ground-truth frame; as a result, this training will not require any labels. We also experimented with an additional adversarial cost as in Mathieu et al. (2015) but did not observe any improvement for the stability prediction task. We hypothesize that although the adversarial objective function helps to have sharper images, such improved sample quality does not transfer to better stability prediction. Figure 1 shows a few examples of the generated data on the test set. Mean squared error is minimized using the AdaM Optimizer(Kingma & Ba, 2014) and we use earlystopping when the validation loss does not improve for 100 epochs.
We extend this ConvDeconv model in a second architecture, named ConvLSTMDeconv, to predict the next frame at each timestep. This model is composed of an LSTM architecture. The same convolutional and deconvolutional blocks as ConvDeconv is utilized to respectively input the current frame to the LSTM transition and output the next frame from the current LSTM state. The details of the ConvLSTMDeconv model architecture are shown in Table 2 and Figure 3 shows the diagram of the both architectures. During the training at each time step the ground-truth data feeds in to the model, but during the test time only the initial time step gets the first frame from the data and for subsequent time steps the generated frames from the previous time steps feed in to the model. The is similar setup to recurrent neural network language models Mikolov (2012), and this is necessary as during the test time we only have access to the first frame. As before, the model is trained to predict the next frame at each time step by minimizing the predictive mean-squared-error using AdaM optimizer and early-stopping. For training, we further subsample in time dimension and reduce the sequence length to 5-time steps. Figure 2 shows some sample generated sequences from the test set.
Layer Type Output channels/Dimension Kernel/Pool size
1 Conv 64 3× 3 2 MaxPool 64 4× 4 3 Conv 128 3× 3 4 MaxPool 64 3× 3 5 Conv 64 3× 3 6 MaxPool 64 3× 3 7 FC LSTM 2000 8 FC 64× 64× 3 9 DeConv 64 3× 3 10 DeConv 64 3× 3 11 DeConv 3 3× 3
Table 2: ConvLSTMDeconv model architecture. FC stands for ”Fully Connected”.
4.2 STABILITY PREDICTION
We have two supervised models for stability prediction. The first one will be a baseline that takes as input the first frame and predict the fall of the tower. For this model we use 50 layer ResNet
architecture from He et al. (2016). We trained the baseline model on each of the different tower heights 3, 4, 5. We call it the single model and name experiments 3S, 4S, 5S respectively for the number of blocks that it was trained on. The second model will be the one using the generated data: it takes as input the first frame and the generated last frame. It consisted of two 50 Layer ResNet blocks in parallel, one for the first frame and one for last frame and the last hidden layer of both models are concatenated together before a logistic regression layer (or Softmax in the case of non-binary labels). Both ResNet blocks share parameters. Based on whether the generated data is coming from ConvDeconv model or ConvLSTMDeconv model we labeled experiments as 3CD, 4CD, 5CD and 3CLD, 4CLD, 5CLD respectively.
None of the models are pre-trained and all the weights are randomly initialized. As in 4.1, we use AdaM and we stopped the training when the validation accuracy was not improved for 100 epochs. All images are contrast normalized independently and we augment our training set using random horizontal flip of the images and randomly changing the contrast and brightness.
5 RESULTS
Figure 4 shows the classification results for each of the 9 models described in Section 4.2 tested on 3, 4 and 5 blocks. Each test case is shown with a different color. And Table 3 shows all the 27 test case results’ numerical values. In almost all cases the generated data improves the generalization performance to test cases with a different number of blocks than it was trained on. For comparison we have included results from Zhang et al. (2016) in Table 4. Since Zhang et al. (2016) only reports the results when the models are trained on tower of 4 blocks, the corresponding results would be the second block row in Table 3, models 4S, 4CD and 4CLD. Even though the datasets are not the same, but it can be observed that the range of performance of the baseline 4S model is consistent with the range of performance of AlexNet model on Table 4. It can be seen that how the results of the 4CD model are significantly better than both IPE and human performance reported in Zhang et al. (2016), while the baselines have similar performances.
One observation is the fact that the improvements are more significant when it’s been tested on scenarios with more bricks than during training. It also improves the reverse case, i.e. fewer bricks than during training, but the improvement is not as significant. It is worth mentioning that testing on a lower number of bricks is a much harder problem as pointed out in Zhang et al. (2016) too. In their case, the prediction performance was almost random when going from 4 blocks to 3 blocks, which is not the case in our experiments2. One possible explanation for performance loss is that a balanced tower with fewer blocks corresponds to an unstable configuration for a tower with more blocks e.g. a tower with 3 blocks is classified as unstable for a prediction model trained on towers of 5 blocks. One solution could be to train these models to predict how many blocks have fallen instead of a binary stability label. Because we have access to this data in our dataset, we explored the same experiments using these labels. Unfortunately, we did not observe any significant improvement. The main reason could be that the distribution of the number of fallen blocks is extremely unbalanced. It is hard to collect data with a balanced number of fallen blocks because some configurations are thus very unlikely e.g. a tower of 5 blocks with only two blocks falls (the majority of the time the whole tower collapses).
The another observation is the fact that models that use ConvDeconv generated data performed slightly better than those that use ConvLSTMDeconv. As seen in Figure 2 the samples in the ConvLSTMDeconv case are more noisy and less sharper than those in Figure 1. This could be caused since after the first time step the model outputs from the last time step is used as input for the next time step, the samples degenerates the longer the sequence is.
Data augmentation was crucial to increase the generalization performance of the stability prediction e.g. 5CD model tested on 4 bricks achieved only 50% without data augmentation while reaching 74.5% accuracy with data augmentation. This significant improvement from data augmentation could be partly because our dataset was relatively small.
2We are not using the same dataset as Zhang et al. (2016) and hence direct comparison is not possible.
Model Train set Test set Accuracy AlexNet 4 3 51 % AlexNet 4 4 95 % AlexNet 4 5 78.5 %
IPE N/A 3 72 % IPE N/A 4 64 % IPE N/A 5 56 %
Human N/A 3 76.5 % Human N/A 4 68.5 % Human N/A 5 59 %
Table 4: The results reported on Zhang et al. (2016). We emphasize that these results are on a different dataset.
6 CONCLUSION
In this paper, we showed that data generated from an unsupervised model could help a supervised learner to generalize to unseen scenarios. We argue that this ability of transfer learning and generalization by observing the world could be one of the ingredients to construct a model of the world that could have applications in many tasks, such as model-based RL. We aim to extend this work in future by looking at the videos of robots manipulating objects and being able to predict their failure beforehand, which could help an RL agent to explore more intelligently.
ACKNOWLEDGMENTS
We would like to thank Harm de Vries and Laurent Dinh for their help and feedback in writing the paper. And also thank Adam Lerer and Jiajun Wu for sharing their dataset. We thank NSERC, CIFAR, IBM, Canada Research Chairs, Google and Samsung for funding.

Summary
===
This paper trains models to predict whether block towers will fall down
or not. It shows that an additional model of how blocks fall down
(predicting a sequence of frames via unsupervised learning) helps the original
supervised task to generalize better.
This work constructs a synthetic dataset of block towers containing
3 to 5 blocks places in more or less precarious positions. It includes both
labels (the tower falls or not) and video frame sequences of the tower's
evolution according to a physics engine.
Three kinds of models are trained. The first (S) simply takes an image of a
tower's starting state and predicts whether it will fall or not. The
other two types (CD and CLD) take both the start state and the final state of the
tower (after it has or has not fallen) and predict whether it has fallen or not,
they only differ in how the final state is provided. One model (ConvDeconv, CD)
predicts the final frame using only the start frame and the other
(ConvLSTMDeconv) predicts a series of intermediate frames before coming
to the final frame. Both CD and CLD are unsupervised.
Each model is trained on towers of a particular heigh and tested on
towers with an unseen height. When the height of the train towers
is the same as the test tower height, all models perform roughly the same
(with in a few percentage points). However, when the test height is
greater than the train height it is extremely helpful to explicitly
model the final state of the block tower before deciding whether it has
fallen or not (via CD and CLD models).
Pros
===
* There are very clear (large) gains in accuracy from adding an unsupervised
final frame predictor. Because the generalization problem is also particularly
clear (train and test with different numbers of blocks), this makes for
a very nice toy example where unsupervised learning provides a clear benefit.
* The writing is clear.
Cons
===
My one major concern is a lack of more detailed analysis. The paper
establishes a base result, but does not explore the idea to the extent
to which I think an ICLR paper should. Two general directions for potential
analysis follow:
* Is this a limitation of the particular way the block towers are rendered?
The LSTM model could be limited by the sub-sampling strategy. It looks
like the sampling may be too coarse from the provided examples. For the
two towers in figure 2 that fall, they have fallen after only 1 or 2
time steps. How quickly do most towers fall? What happens if the LSTM
is trained at a higher frame rate? What is the frame-by-frame video
prediction accuracy of the LSTM? (Is that quantity meaningful?)
How much does performance improve if the LSTM is provided ground truth
for only the first k frames?
* Why is generalization to different block heights limited?
Is it limited by model capacity or architecture design?
What would happen if the S-type models were made wider/deeper with the CD/CLD
fall predictor capacity fixed?
Is it limited by the precise task specification?
What would happen if networks were trained with towers of multiple heights
(apparently this experiment is in the works)?
I appreciate that one experiment in this direction was provided.
Is it limited by training procedure? What if the CD/CLD models were trained
in an end-to-end manner? What if the double frame fall predictor were trained
with ground truth final frames instead of generated final frames?
Minor concerns:
* It may be asking too much to re-implement Zhang et. al. 2016 and PhysNet
for the newly proposed dataset, but it would help the paper to have baselines
which are directly comparable to the proposed results. I do not think this
is a major concern because the point of the paper is about the role of
unsupervised learning rather than creating the best fall prediction network.
* The auxiliary experiment provided is motivated as follows:
"One solution could be to train these models to predict how many blocks have
fallen instead of a binary stability label."
Is there a clear intuition for why this might make the task easier?
* Will the dataset, or code to generate it, be released?
Overall Evaluation
===
The writing, presentation, and experiments are clear and of high enough
quality for ICLR. However the experiments provide limited analysis past
the main result (see comments above). The idea is a clear extension of ideas behind unsupervised
learning (video prediction) and recent results in intuitive physics from
Lerer et. al. 2016 and Zhang et. al. 2016, so there is only moderate novelty.
However, these results would provide a valuable addition to the literation,
especially if more analysis was provided.