Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.
1 INTRODUCTION
Research and application of recurrent neural networks (RNNs) have seen explosive growth over the last few years, (Martens & Sutskever, 2011; Graves et al., 2009), and RNNs have become the central component for some very successful model classes and application domains in deep learning (speech recognition (Amodei et al., 2015), seq2seq (Sutskever et al., 2014), neural machine translation (Bahdanau et al., 2014), the DRAW model (Gregor et al., 2015), educational applications (Piech et al., 2015), and scientific discovery (Mante et al., 2013)). Despite these recent successes, it is widely acknowledged that designing and training the RNN components in complex models can be extremely tricky. Painfully acquired RNN expertise is still crucial to the success of most projects.
One of the main strategies involved in the deployment of RNN models is the use of the Long Short Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997), and more recently the Gated Recurrent Unit (GRU) proposed by Cho et al. (2014); Chung et al. (2014) (we refer to these as gated architectures). The resulting models are perceived as being more easily trained, and achieving lower error. While it is widely appreciated that RNNs are universal approximators (Doya, 1993), an unresolved question is the degree to which gated models are more computationally powerful in practice, as opposed to simply being easier to train.
Here we provide evidence that the observed superiority of gated models over vanilla RNN models is almost exclusively driven by trainability. First we describe two types of capacity bottlenecks that various RNN architectures might be expected to suffer from: parameter efficiency related to learning the task, and the ability to remember input history. Next, we describe our experimental setup where we disentangle the effects of these two bottlenecks, including training with extremely thorough hyperparameter (HP) optimization. Finally, we describe our capacity experiment results
∗Work done as a member of the Google Brain Residency program (g.co/brainresidency).
(per-parameter and per-unit), as well as the results of trainability experiments (training on extremely hard tasks where gated models might reasonably be expected to perform better).
1.1 CAPACITY BOTTLENECKS
There are several potential bottlenecks for RNNs, for example: How much information about the task can they store in their parameters? How much information about the input history can they store in their units? These first two bottlenecks can both be seen as memory capacities (one for the task, one for the inputs), for different types of memory.
Another, different kind of capacity stems from the set of computational primitives an RNN is able to perform. For example, maybe one wants to multiply two numbers. In terms of number of units and time steps, this task may be very straight-forward using some specific computational primitives and dynamics, but with others it may be extremely resource heavy. One might expect that differences in computational capacity due to different computational primitives would play a large role in performance. However, despite the fact that the gated architectures are outfitted with a multiplicative primitive between hidden units, while the vanilla RNN is not, we found no evidence of a computational bottleneck in our experiments. We therefore will focus only on the per-parameter capacity of an RNN to learn about its task during training, and on the per-unit memory capacity of an RNN to remember its inputs.
1.2 EXPERIMENTAL SETUP
RNNs have many HPs, such as the scalings of matrices and biases, and the functional form of certain nonlinearities. There are additionally many HPs involved in training, such as the choice of optimizer, and the learning rate schedule. In order to train our models we employed a HP tuner that uses a Gaussian Process model similar to Spearmint (see Appendix, section on HP tuning and Desautels et al. (2014); Snoek et al. (2012) for related work). The basic idea is that one requests HP values from the tuner, runs the optimization to completion using those values, and then returns the validation loss. This loss is then used by the tuner, in combination with previously reported losses, to choose new HP values such that over many experiments, the validation loss is minimized with respect to the HPs. For our experiments, we report the evaluation loss (separate from the validation loss returned to the HP optimizer, except where otherwise noted) after the HP tuner has highly optimized the task (hundreds to many thousands of experiments for each architecture and task).
In our studies we used a variety of well-known RNN architectures: standard RNNs such as the vanilla RNN and the newer IRNN (Le et al., 2015), as well as gated RNN architectures such as the GRU and LSTM. We rounded out our set of models by innovating two novel (to our knowledge) RNN architectures (see Section 1.4) we call the Update Gate RNN (UGRNN), and the Intersection RNN (+RNN). The UGRNN is a ‘minimally gated’ RNN architecture that has only a coupled gate between the recurrent hidden state, and the update to the hidden state. The +RNN uses coupled gates to gate both the recurrent and depth dimensions in a straightforward way.
To further explore the various strengths and weaknesses of each RNN architecture, we also used a variety of network depths: 1, 2, 4, 8, in our experiments.1 In most experiments, we held the number of parameters fixed across different architectures and different depths. More precisely, for a given experiment, a maximum number of parameters was set, along with an input and output dimension. The number of hidden units per layer was then chosen such that the number of parameters, summed across all layers of the network, was as large as possible without exceeding the allowed maximum.
For each of our 6 tasks, 6 RNN variants, 4 depths, and 6+ model sizes, we ran the HP tuner in order to optimize the relevant loss function. Typically this resulted in many hundreds to several thousands of HP evaluations, each of which was a full training run up to millions of training steps. Taken together, this amounted to CPU-millennia worth of computation.
1.3 RELATED WORK
While it is well known that RNNs are universal approximators of arbitrary dynamical systems (Doya, 1993), there is little theoretical work on the task-capacity of RNNs. Koiran & Sontag (1998) studied
1Not all experiments used a depth of 8, due to limits on computational resources.
the VC dimension of RNNs, which provides an upper bound on their task-capacity (defined in Section 2.1). These upper bounds are not a close match to our experimental results. For instance, we find that performance saturates rapidly in terms of the number of unrolling steps (Figure 2b), while the relevant bound increases linearly with the number of unrolling steps. "Unrolling" refers to recurrent computation through time.
Empirically, Karpathy et al. (2015) have studied how LSTMs encode information in character-based text modeling tasks. Further, Sussillo & Barak (2013) have reverse-engineered the vanilla RNN trained on simple tasks, using the tools and language of nonlinear dynamical systems theory. In Foerster et al. (2016) the behavior of switched affine recurrent networks is carefully examined.
The ability of RNNs to store information about their input has been better studied, in both the context of machine learning and theoretical neuroscience. Previous work on short term memory traces explores the tradeoffs between memory fidelity and duration, for the case that a new input is presented to the RNN at every time step (Jaeger & Haas, 2004; Maass et al., 2002; White et al., 2004; Ganguli et al., 2008; Charles et al., 2014). We use a simpler capacity measure consisting only of the ability of an RNN to store a single input vector. Our results suggest that, contrary to common belief, the capacity of RNNs to remember their input history is not a practical limiting factor on their performance.
The precise details of what makes an RNN architecture perform well is an extremely active research field (e.g. Jozefowicz et al. (2015)). A highly related article is Greff et al. (2015), in which the authors used random search of HPs, along with systematic removal of pieces of the LSTM architecture to determine which pieces of the LSTM were more important than the others. Our UGRNN architecture is directly inspired by the large impact of removing the forget gate from the LSTM (Gers et al., 1999). Zhou et al. (2016) introduced an architecture with minimal gating that is similar to the UGRNN, but is directly inspired by the GRU. An in-depth comparison between RNNs and GRUs in the context of end-to-end speech recognition and a limited computational budget was conducted in Amodei et al. (2015). Further, ideas from RNN architectures that improve ease of training, such as forget gates (Gers et al., 1999), and copying recurrent state from one time step to another, are making their way into deep feed-forward networks as highway networks (Srivastava et al., 2015) and residual connections (He et al., 2015), respectively. Indeed, the +RNN was inspired in part by the coupled depth gate of Srivastava et al. (2015).
1.4 RECURRENT NEURAL NETWORK ARCHITECTURES
Below we briefly define the RNN architectures used in this study. Unless otherwise stated W denotes a matrix, b denotes a vector of biases. The symbol xt is the input at time t, and ht is the hidden state at time t. Remaining vector variables represent intermediate values. The function σ(·) denotes the logistic sigmoid function and s(·) is either tanh or ReLU, set as a HP (see Appendix, Section RNN HPs for the complete list of HPs). Initial conditions for the networks were set to a learned bias. Finally, it is a well-known trick of the trade to initialize the gates of an LSTM or GRU with a large bias to induce better gradient flow. We included this parameter, denoted as bfg, and tuned it along with all other HPs.
RNN, IRNN (LE ET AL., 2015)
ht = s ( Whht−1 +W xxt + b h )
(1)
Note the IRNN is identical in structure to the vanilla RNN, but with an identity initialization for Wh, zero initialization for the biases, and s = ReLU only.
UGRNN - UPDATE GATE RNN
Based on Greff et al. (2015), where they noticed the forget gate “was crucial” to LSTM performance, we tried an RNN variant where we began with a vanilla RNN and added a single gate. This gate determines whether the hidden state is carried over from the previous time step, or updated – hence, it is an update gate. An alternative way to view the UGRNN is a highway layer gated through time
(Srivastava et al., 2015). ct = s ( Wch ht−1 +W cx xt + b c )
(2) gt = σ ( Wgh ht−1 +W gx xt + b g + bfg ) (3)
ht = gt · ht−1 + (1− gt) · ct (4)
GRU - GATED RECURRENT UNIT (CHO ET AL., 2014) rt = σ ( Wrh ht−1 +W rx xt + b r )
(5) ut = σ ( Wuh ht−1 +W ux xt + b u + bfg ) (6)
ct = s ( Wch (rt · ht−1) +Wcx xt + bc ) (7)
ht = ut · ht−1 + (1− ut) · ct (8)
LSTM - LONG SHORT TERM MEMORY(HOCHREITER & SCHMIDHUBER, 1997) it = σ ( Wih ht−1 +W ix xt + b i )
(9) ft = σ ( Wfh ht−1 +W fx xt + b f + bfg ) (10)
cint = s ( Wch ht−1 +W cx xt + b c )
(11)
ct = ft · ct−1 + it · cint (12) ot = σ ( Woh ht−1 +W ox xt + b o ) (13)
ht = ot · tanh(ct) (14)
+RNN - INTERSECTION RNN
Due to the success of the UGRNN for shallower architectures in this study (see later figures on trainability), as well as some of the observed trainability problems for both the LSTM and GRU for deeper architectures (e.g. Figure 4h) we developed the Intersection RNN (denoted with a ‘+’) architecture with a coupled depth gate in addition to a coupled recurrent gate. Additional influences for this architecture were the recurrent gating of the LSTM and GRU, and the depth gating from the highway network (Srivastava et al., 2015). This architecture has recurrent input, ht−1, and depth input, xt. It also has recurrent output, ht, and depth output, yt. Note that this architecture only applies between layers where xt and yt have the same dimension, and is not appropriate for networks with a depth of 1 (we exclude depth one +RNNs in our experiments).
yint = s1 (W yh ht−1 +W yx xt + b y) (15)
hint = s2 (W hh ht−1 +W hx xt + b h) (16) gyt = σ ( Wg yh ht−1 +W gyx xt + b gy + bfg,y ) (17)
ght = σ ( Wg hh ht−1 +W ghx xt + b gh + bfg,h )
(18)
yt = g y t · xt + (1− g y t ) · yint (19)
ht = g h t · ht−1 + (1− ght ) · hint (20)
In practice we used ReLU for s1 and tanh for s2.
2 CAPACITY EXPERIMENTS
2.1 PER-PARAMETER CAPACITY
A foundational result in machine learning is that a single-layer perceptron with N2 parameters can store at least 2 bits of information per parameter (Cover, 1965; Gardner, 1988; Baldi & Venkatesh, 1987). More precisely, a perceptron can implement a mapping from 2N ,N -dimensional, input vectors to arbitrary N -dimensional binary output vectors, subject only to the extremely weak restriction that the input vectors be in general position. RNNs provide a far more complex input-output mapping, with hidden units, recurrent dynamics, and a diversity of nonlinearities. Nonetheless, we wondered if there were analogous capacity results for RNNs that we might be able to observe empirically.
2.1.1 EXPERIMENTAL SETUP
As we will show in Section 3, tasks with complex temporal dynamics, such as language modeling, exhibit a per-parameter capacity bottleneck that explains the performance of RNNs far better than a per-unit bottleneck. To make the experimental design as simple as possible, and to remove potential confounds stemming from the choice of temporal dynamics, we study per-parameter capacity using a task inspired by Gardner (1988). Specifically, to measure how much task-related information can be stored in the parameters of an RNN, we use a memorization task, where a random static input is injected into an RNN, and a random static output is read out some number of time steps later. We emphasize that the same per-parameter bottleneck that we find in this simplified task also arises in more temporally complex tasks, such as language modeling.
At a high level, we draw a fixed set of random inputs and random labels, and train the RNN to map random inputs to randomly chosen labels via cross-entropy error. However, rather than returning the cross-entropy error to the HP tuner (as is normally done), we instead return the mutual information between the RNN outputs and the true labels. In this way, we can treat the number of input-output mappings as a HP, and the tuner will select for us the correct number of mappings so as to maximize the mutual information between the RNN outputs and the labels. From this mutual information we compute bits per parameter, which provides a normalized measurement of how much the RNN learned about the task.
More precisely, we draw datasets of binary inputs X and target binary labels Y at uniform from the set of all binary datasets, X ∼ X = {0, 1}nin×b, Y ∼ Y = {0, 1}1×b, where b is the number of samples, and nin is the dimensionality of the inputs. Number of samples, b, is treated as a HP and in practice the optimal dataset size is very close to the bits of mutual information between true and predicted labels. This trend is demonstrated in Figure App.1 in the Appendix. For each value of b the RNN is trained to minimize the cross entropy of the network output with the true labels. We write the output of the RNN for all inputs as Ŷ = f (X), with corresponding random variable Ŷ . We are interested in the mutual information I ( Y; Ŷ ) between the true class labels and the class labels
predicted by the RNN. This is the amount of (directly recoverable) information that the RNN has stored about the task. In this setting, it is calculated as
I ( Y; Ŷ ) = H (Y)−H ( Y|Ŷ ) (21)
= b+ b (p log2 p+ (1− p) log2 (1− p)) , (22)
where p is the fraction of correctly classified samples. The number b is then adjusted, along with all the other HPs, so as to maximize the mutual information I ( Y; Ŷ ) . In practice p is computed using
only a single draw of {X,Y} ∼ X × Y . We performed this optimization of I ( Y; Ŷ ) for various RNN architectures, depths, and numbers of
parameters. We plot the best value of I ( Y; Ŷ ) vs. number of parameters in Figure 1a. This captures
the amount of information stored in the parameters about the mapping between X and Y. To get an estimate of bits per parameter, we divide by the number of parameters, as shown in Figure 1e.
2.1.2 RESULTS
Five Bits per Parameter Examining the results of Figure 1, we find the capacity of all architectures is roughly linear in the number of parameters, across several orders of magnitude of parameter count. We further find that the capacity is between 3 and 6 bits per parameter, once again across all architectures, depths 1, 2 and 4, and across several orders of magnitude in terms of number of parameters. Given the possibility of small size effects, and a larger portion of weights used as biases at a small number of parameters, we believe our estimates for larger networks are more reliable. This leads us to a bits per parameter estimate of approximately 5, averaging over all architectures and all depths. Finally, we note that the per-parameter task capacity increases as a function of the number of unrollings, though with diminishing gains (Figure 2b).
The finding that our results are consistent across diverse architectures and scales is even more surprising, since prior to these experiments it was not clear that capacity would even scale linearly
with the number of parameters. For instance, previous results on model compression – by reducing the number of parameters (Yang et al., 2015), or by reducing the bit depth of parameters (Hubara et al., 2016) – might lead one to predict that different architectures use parameters with vastly different efficiencies, and that task capacity increases only sublinearly with parameter count.
Gating Slightly Reduces Capacity While overall, the different architectures performed very similarly, there are some capacity differences between architectures that appear to hold up across most depths and parameter counts. To quantify these differences we constructed a table showing the change in the number of parameters one would need to switch from one architecture to another, while maintaining equivalent capacity (Figure 1i). One trend that emerged from our capacity experiments is a slightly reduced capacity as a function of "gatedness". Putting aside the IRNN, which performed the worst and is discussed below, we noticed that across all depths and all model sizes, the performance was on average RNN > UGRNN > GRU > LSTM > +RNN. The vanilla RNN has no gates, the UGRNN has one, while the remaining three have two or more.
ReLUs Reduce Capacity In our capacity tasks, the IRNN performed noticeably worse than all other architectures, reaching a maximum bits per parameter of roughly 3.5. To determine if this performance drop was due to the ReLU nonlinearity of the IRNN, or its identity initialization, we sorted through the RNN and UGRNN results (which both have ReLU and tanh as choices for the nonlinearity HP) and looked at the maximum bits per parameter when only optimizations using ReLU
are considered. Indeed, both the RNN and UGRNN bits per parameter dropped dramatically to the 3.5 range (Figure 2a) when those architectures exclusively used ReLU, providing strong evidence that the ReLU activation function is problematic for this capacity task.
2.2 PER-UNIT CAPACITY TO REMEMBER INPUTS
An additional capacity bottleneck in RNNs is their ability to store information about their inputs over time. It may be plainly obvious that an IRNN, which is essentially an integrator, can achieve perfect memory of its inputs if the number of inputs is less than or equal to the number of hidden units, but it is not so clear for some of the more complex architectures. So we measured the per-unit input memory empirically. Figure 2c shows the intuitive result that every RNN architecture (at every depth and number of parameters) we studied can reconstruct a random nin dimensional input at some time in the future, if and only if the number of hidden units per layer in the network, nh, is greater than or equal to nin Moreover, regardless of RNN architecture, the error in reconstructing the input follows the same curve as a function of the number of hidden units for all RNN variants, corresponding to reconstructing an nh dimensional subspace of the nin dimensional input.
We highlight this per-unit capacity to make the point that a per-parameter task capacity appears to be the limiting factor in our experiments (e.g. Figure 1 and Figure 3), and not a per-unit capacity, such as the per-unit capacity to remember previous inputs. Thus when comparing results between architectures, one should normalize different architectures by the number of parameters, and not the number of units, as is frequently done in the literature (e.g. when comparing vanilla RNNs to LSTMs). This makes further sense as, for all common RNN architectures, the computational cost of processing a single sample is linear in the number of parameters, and quadratic in the number of units per layer. As we show in Figure 3d, plotting the capacity results by numbers of units gives very misleading results.
3 ADDITIONAL TASKS WHERE ARCHITECTURES ACHIEVE VERY SIMILAR LOSS
We studied additional tasks that we believed to be easy enough to train that the evaluation loss of different architectures would reveal variations in capacity rather than trainability. A critical aspect of these tasks is that they could not be learned perfectly by any of the model sizes in our experiments. As we change model size, we therefore expect performance on the task to also change. The tasks are (see Appendix, section Task Definitions for further elaboration of these tasks):
• text8 - 1-step ahead character-based prediction on the text8 Wikipedia dataset (100 million characters) (Mahoney, 2011). • Random Continuous Functions (RCF) - A task similar to the per-parameter capacity task
above, except the target outputs are real numbers (not categorical), and the number of training samples is held fixed.
The performance on these two tasks is shown in Figure 3. The evaluation loss as a function of the number of parameters is plotted in panels a-c and e-g, for the text8 task, and RCF task, respectively. For all tasks in this section, the number of parameters rather than the number of units provided the bottleneck on performance, and all architectures performed extremely closely for the same number of parameters. By close performance we mean that, for one model to achieve the same loss as another the model, the number of parameters would have to be adjusted by only a small factor (exemplified in Figure 1i for the per-parameter capacity task).
4 TASKS THAT ARE VERY HARD TO LEARN
In practice it is widely appreciated that there is often a significant gap in performance between, for example, the LSTM and the vanilla RNN, with the LSTM nearly always outperforming the vanilla RNN. Our per-parameter capacity results provide evidence for a rough equivalence among a variety of RNN architectures, with slightly higher capacity in the vanilla RNN (Figure 1). To reconcile our per-parameter capacity results with widely held experience, we provide evidence that gated architectures, such as the LSTM, are far easier to train than the vanilla RNN (and often the IRNN).
We study two tasks that are difficult to learn: parallel parentheses counting of independent input streams, and mathematical addition of integers encoded in a character string (see Appendix, section Task Definitions). The parentheses task is moderately difficult to learn, while the arithmetic task is quite hard. The results of the HP optimizations are shown in Figure 4a-4h for the parentheses task, and in Figure 4i-4p for the arithmetic task. These tasks show that, while it is possible for a vanilla RNN to learn these tasks reasonably well, it is far more difficult than for a gated architecture. Note that the best achieved loss on the arithmetic task is still significantly decreasing, even after 2500 HP evaluations (2500 full complete optimizations over the training set), for the RNN and IRNN.
There are three noteworthy trends in these trainability experiments. First, across both tasks, and all depths (1, 2, 4 and 8), the RNN and IRNN performed most poorly, and took the longest to learn the task. Note, however that both the RNN and IRNN always solved the tasks eventually, at least for depth 1. Second, as the stacking depth increased, the gated architectures became the only architectures that
could solve the tasks. Third, the most trainable architecture for depth 1 was the GRU, and the most trainable architecture for depth 8 was the +RNN (which performed the best on both of our metrics for trainability, on both tasks).
To achieve our results on capacity and trainability, we relied heavily on a HP tuner. Most practitioners do not have the time or resources to make use of such a tuner, typically only adjusting the HPs a few times themselves. So we wondered how the various architectures would perform if we set HPs randomly, within the ranges specified (see Appendix for ranges). We tried this 1000 times on the parentheses task, for all 200k parameter architectures at depths 1 and 8 (Figure 5 and Table 1). The noticeable trends are that the IRNN returned an infeasible error nearly half of the time, and the LSTM
(depth 1) and GRU (depth 8) were infeasible the least number of times, where infeasibility means that the training loss diverged. For depth 1, the GRU gave the smallest error, and the smallest median error, and for depth 8, the +RNN delivered the smallest error and smallest median error.
5 DISCUSSION
Here we report that a number of RNN variants can hold between 3-6 bits per parameter about their task, and that these variants can remember a number of random inputs that is nearly equal to the number of hidden units in the RNN. The quantification of the number of bits per parameter an RNN can store about a task is particularly important, as it was not previously known whether the amount of information about a task that could be stored was even linear in the number of parameters.
While our results point to empirical capacity limits for both task memorization, and input memorization, apparently the requirement to remember features of the input through time is not a practical bottleneck. If it were, then the vanilla RNN and IRNN would perform better than the gated architectures in proportion to the ratio of the number of units, which they do not. Based on widespread results in the literature, and our own results on our difficult tasks, the loss of some memory capacity (and possibly a small amount of per-parameter storage capacity) for improved trainability seems a worthwhile trade off. Indeed, the input memory capacity did not obviously impact any task not explicitly designed to measure it, as the error curves – for instance for the language modeling task – overlapped across architectures for the same number of parameters, but not the same number of units.
Our result on per-parameter task capacity, about 5 bits per parameter averaged over architectures, is in surprising agreement with recently published results on the capacity of synapses in biological neurons. This number was recently calculated to be about 4.7 bits per synapse, based on biological synapses in the hippocampus having roughly 26 measurable discrete sizes (Bartol et al., 2016). Our capacity results have implications for compressed networks that employ quantization techniques. In
particular, they provide an estimate of the number of bits which a weight may be compressed without loss in task performance. Coincidentally, in Han et al. (2015), the authors used 5 bits per weight in the fully connected layers.
An additional observation about per-parameter task capacity in our experiments is that it increases for a few time steps beyond one (Figure 2b), and then appears to saturate. We interpret this to suggest that recurrence endows additional capacity to a network with shared parameters, but that there are diminishing returns, and the total capacity remains bounded even as the number of time steps increases.
We also note that performance is nearly constant across RNN architectures if the number of parameters is held fixed. This may motivate the design and use of architectures with small compute per parameter ratios, such as mixture of experts RNNs (Shazeer et al., 2017), and RNNs with large embedding dictionaries on input and output (Józefowicz et al., 2016).
Despite our best efforts, we cannot claim that we perfectly trained any of the models. Potential problems in HP optimization could be local minima, as well as stochastic behavior in the HP optimization as a result of the stochasticity of batching or random draws for weight matrices. We tried to uncover these effects by running the best performing HPs 100 times, and did not observe any serious deviations from the best results (see Table App.1 in Appendix). Another form of validation comes from the fact that in our capacity task, essentially 3 independent experiments (one for each level of depth) yielded a clustering by architecture (Figure 1e).
Do our results yield a framework for choosing a recurrent architecture? In total, we believe yes. As explored in Amodei et al. (2015), a practical concern for recurrent models is speed of execution in a production environment. Our results suggest that if one has a large resource budget for training and confined resource budget for inference, one should choose the vanilla RNN. Conversely, if the training resource budget is small, but the inference budget large, one should choose a gated model. Another serious concern relates to task complexity. If the task is easy to learn, a vanilla RNN should yield good results. However if the task is even moderately difficult to learn, a gated architecture is the right choice. Our results point to the GRU as being the most learnable of gated RNNs for shallow architectures, followed by the UGRNN. The +RNN typically performed best for deeper architectures. Our results on trainability confirm the widely held view that the LSTM is an extremely reliable architecture, but it was almost never the best performer in our experiments. Of course further experiments will be required to fully vet the UGRNN and +RNN. All things considered, in an uncertain training environment, our results suggest using the GRU or +RNN.
6 ACKNOWLEDGEMENTS
We would like to thank Geoffrey Irving, Alex Alemi, Quoc Le, Navdeep Jaitly, and Taco Cohen for helpful feedback.
Appendix
A RNN HPS SET BY THE HP TUNER
We used a HP tuner that uses a Gaussian Process (GP) Bandits approach for HP optimization. Our setting of the tuner’s internal parameters was such that it uses Batched GP Bandits with an expected improvement acquisition function and a Matern 5/2 Kernel with feature scaling and automatic relevance determination performed by optimizing over kernel HPs. Please see Desautels et al. (2014) and Snoek et al. (2012) for closely related work.
For all our tasks, we requested HPs from the tuner, and reported loss on a validation dataset. For the per-parameter capacity task, the evaluation, validation and training datasets were identical. For text8, the validation and evaluation sets consisted of different sections of held out data. For all other tasks, evaluation, validation, and training sets were randomly drawn from the same distribution. The performance we plot in all cases is on the evaluation dataset.
Below is the list of all tunable HPs that were generically applied to all models. In total, each RNN variant had between 10 and 27 HP dimensions relating to the architecture, optimization, and regularization.
• s() - as used in the following RNN definitions, a nonlinearity determined by the HP tuner, {ReLU, tanh}. The only exception was the IRNN, which used ReLU exclusively. • For any matrix that is inherently square, e.g. Wh, there were three possible initializations:
identity, orthogonal, or random normal distribution scaled by 1/ √ nh, with nh the number of recurrent units. The sole exception was the RNN, which was limited to either orthogonal or random normal initializations, to differentiate it from the IRNN. For any matrix that is inherently rectangular, e.g. Wx, we initialized with a random normal distribution scaled by 1/ √ nin, with nin the number of inputs.
• For all matrix initializations except the identity initialization, there was a multiplicative scalar used to set the scale of matrix. The scalar was exponentially distributed in [0.01, 2.0] for recurrent matrices and [0.001, 2.0] for rectangular matrices.
• Biases could have two possible distributions: all biases set to a constant value, or drawn from a standard normal distribution. • For all bias initializations, a multiplicative scalar was drawn, uniformly distributed in [−2.0, 2.0] and applied to bias initialization. • We included a scalar bias HP bfg for architectures that contain forget or update gates, as is
commonly employed in practice, which was uniformly distributed in [0.0, 6.0].
Additionally, the HP tuner was used to optimize HPs associated with learning:
• The number of training steps - The exact range varied between tasks, but always fell between 50K and 20M. • One of four optimization algorithms could be chosen: vanilla SGD, SGD with momentum,
RMSProp (Tieleman & Hinton, 2012), or ADAM (Kingma & Ba, 2014). • learning rate initial value, exponentially distributed in [1e−4, 1e−1] • learning rate decay - exponentially distributed in [1e−3, 1]. The learning rate exponentially
decays by this factor over the number of training steps chosen by the tuner • optimizer momentum-like parameter - expressed as a logit, and uniformly distributed in [1.0, 7.0]
• gradient clipping value - exponentially distributed in [1, 100] • l2 decay - exponentially distributed in [1e−8, 1e−3].
The perceptron capacity task also had associated HPs:
• The number of samples in the dataset, b - between 0.1x and 10x the number of model parameters
• A HP determined whether the input vector X·j was presented to the RNN only at the first time step, or whether it was presented at every time step.
Some optimization algorithms had additional parameters such as ADAM’s second order decay rate, or epsilon parameter. These were set to their default values and not optimized. The batch size was set individually by hand for all experiments. The same seed was used to initialize the random number generator for all task parameters, whereas the generator was randomly seeded for network parameters (e.g. initializations). Note that for each network, the initial condition was set to a learned vector.
Figure App.1: In the capacity task, the optimal dataset size found by the HP tuner was only slightly larger than the mutual information in bits reported in Figure 1a, for all architectures at all sizes and depths.
B TASK DEFINITIONS
PERCEPTRON CAPACITY
While at a high-level, for the perceptron capacity task, we wanted to optimize the amount of information the RNN carried about true random labels, in practice, the training objective was standard cross-entropy. However, when returning a validation loss to the HP tuner, we returned the mutual information I ( Y; Ŷ|X ) . Conceptually, this is as if there is one nested optimization inside another.
The inner loop optimizes the RNN for the set of HPs, training cross entropy, but returning mutual information. The outer loop then chooses the HPs, in particular, the number of samples b, in equation (21), so as to maximize the amount of mutual information. This implementation is necessitated because there is no straightforward way to differentiate mutual information with respect to number of samples. During training, cross entropy error is evaluated beginning after 5 time steps.
MEMORY CAPACITY
In the Memory Capacity task, we wanted to know how much information an RNN can reconstruct about its inputs at some later time point. We picked an input dimension, 64, and varied the number of parameters in the networks such that the number of hidden units was roughly centered around 64. After 12 time steps the target of the network was exact reconstruction of the input, with a square error loss. The inputs were random values drawn from a uniform distribution between − √ 3 and √ 3 (corresponding to a variance of 1).
RANDOM CONTINUOUS FUNCTION
A dataset was constructed consisting of N = 106 random unit norm Gaussian input vectors x, with size d = 50. Target scalar outputs y were generated for each input vector, and were also drawn from a unit norm Gaussian. Each sample i was assigned a power law weighting βi = (i+τ)−1
Z , where Z was a normalization constant such that the weightings summed to 1, and the characteristic time constant τ = 5000. The loss function for training was calculated after 50 time steps and was weighted square error on the yi, with the βi acting as the weighting terms.
TEXT8
In the text8 task, the task was to predict one character ahead in the text8 dataset (1e8 characters of Wikipedia) (Mahoney, 2011). Input was a hot-one encoded sequence, as was the output. The loss was cross-entropy loss on a softmax output layer. Rather than use partial unrolling as is common in language modeling, we generated random pointers into the text. The first 13 time steps (where T = 50) were used to initialize the RNN into a normal operating mode, and remaining steps were used for training or inference.
PARENTHESES COUNTING TASK
The parentheses counting task independently counts the number of opened ‘parens’, e.g. ‘(’, without the closing ‘)’. Here parens is used to mean any of 10 parens type pairs, e.g. ‘<>’ or ‘[]’. Additionally, there were 10 noise characters, ‘a’ to ‘j’. For each paren type, there was a 20D + 10D = 30D hot-one encoding of all paren and noise symbols, for a total of 300 inputs. The output for each paren type was a hot-one encoding of the digits 0-9, which represented the count of the opened parens of that type. If the count exceeded 9, the the network kept the count at 9, if the paren was closed, the count decreased. The loss was the sum of cross-entropy losses, one for each paren type. Finally, for each paren input stream, 50% random noise characters were drawn, and 50% random paren characters were drawn, e.g. 10 streams like ‘(a<a<bcb>[[[)’. Parens of other types were treated as noise for the current type, e.g. for the above string if the paren type was ‘<>’, the answer is ‘1’ at the end. The loss was defined only at the final time point, T , and T = 175.
ARITHMETIC TASK
In the arithmetic task, a hot-one encoded character sequence of an addition problem was presented as input to the network, e.g., ‘-343243+93851= ’, and the output was the hot-one encoded answer, including the correct amount of left padded spaces, ‘-249392’. An additional HP for this task was the number of compute steps (1-6) between the input of the ‘=’ and the first non-space character in the target output sequence. The two numbers in the input were randomly, uniformly selected in [−1e7, 1e7]. After 36 time steps, cross-entropy loss was calculated. We found this task to be extremely difficult for the networks to learn, but when the task was learned, certain of the network architectures could perform the task nearly perfectly.
C HP ROBUSTNESS
We wondered how robust the HPs are to the variability of both random batching of data, and random initialization of parameters. So we identified the best HPs from the parentheses experiments of 100k parameter, 1 layer architectures, and reran the parameter optimization 100 times. We measured the number of infeasible experiments, as well as a number of statistics of the loss for the reruns (Table App.1). These results show that the best HPs yielded a distribution of losses very close to the originally reported loss value.
Architecture Original Infeasible Min Mean Max S.D. S.D./Mean RNN 1.16e-2 0 % 1.41e-2 8.21e-2 0.294 5.22e-2 0.636 IRNN 4.20e-4 48 % 2.24e-4 5.02e-4 8.69e-4 1.35e-4 0.269
UGRNN 1.02e-4 0 % 3.66e-5 2.71e-4 6.06e-3 7.12e-4 2.63 GRU 2.80e-4 1 % 7.66e-5 1.89e-4 5.48e-4 9.08e-5 0.480
LSTM 7.96e-4 0 % 8.10e-4 2.02e-3 0.0145 2.31e-3 1.14
Table App.1: Results of 100 runs on the parentheses task using the best HPs for each architecture, at depth 1. HPs were chosen to be the set which achieved the minimum loss. Table shows original loss achieved by the HP tuner, amount of infeasible trials, minimum loss from running 100 iterations of the same HPs, mean loss, maximum loss, standard deviation, and standard deviation divided by the mean.
1 layer 8 layer t-stat df p-value t-stat df p-value
+RNN/GRU - - - -23.6 1080 < 0.001 +RNN/IRNN - - - -25.7 954 < 0.001 +RNN/LSTM - - - -26.1 941 < 0.001 +RNN/RNN - - - -25.8 946 < 0.001 +RNN/UGRNN - - - -24.3 1050 < 0.001 GRU/IRNN -7.74 696 < 0.001 -3.51 1360 < 0.001 GRU/LSTM -6.65 1750 < 0.001 -4.84 1290 < 0.001 GRU/RNN -26.5 1340 < 0.001 -3.93 1330 < 0.001
GRU/UGRNN -4.11 1620 < 0.001 -1.13 1840 0.261 IRNN/LSTM 2.23 652 0.0264 -2.04 1250 0.0420 IRNN/RNN -12.7 426 < 0.001 -0.571 1250 0.568 IRNN/UGRNN 4.03 719 < 0.001 2.37 1320 0.0178 LSTM/RNN -19.6 1500 < 0.001 1.53 1730 0.125 LSTM/UGRNN 2.25 1640 0.0247 3.81 1260 < 0.001 RNN/UGRNN 20.7 1210 < 0.001 2.81 1300 0.00498
Table App.2: Results of Welch’s t-test for equality of means on evaluation losses of architecture pairs trained on the parentheses task with randomly sampled HPs. 8 layer GRU and UGRNN, IRNN and RNN, and LSTM and RNN pairs have loss distributions that are different with statistical significance (p > 0.05). Negative t-statistic indicates that the mean of the second architecture in the pair is larger than the first.

CONTRIBUTIONS
Large-scale experiments are used to measure the capacity and trainability of different RNN architectures. Capacity experiments suggest that across all architectures, RNNs can store between three and six bits of information per parameter, with ungated RNNs having the highest per-parameter capacity. All architectures are able to store approximately one floating point number per hidden unit. Trainability experiments show that ungated architectures (RNN, IRNN) are much harder to train than gated architectures (GRU, LSTM, UGRNN, +RNN). The paper also proposes two novel RNN architectures (UGRNN and +RNN); experiments suggest that the UGRNN has similar per-parameter capacity as the ungated RNN but is much easier to train, and that deep (8-layer) +RNN models are easier to train than existing architectures.
CLARITY
The paper is well-written and easy to follow.
NOVELTY
This paper is the first to my knowledge to empirically measure the number of bits of information that can be stored per learnable parameter. The idea of measuring network capacity by finding the dataset size and other hyperparameters that maximizes mutual information is a particularly novel experimental setup.
The proposed UGRNN is similar but not identical to the minimal gated unit proposed by Zhou et al, “Minimal Gated Unit for Recurrent Neural Networks”, International Journal of Automation and Computing, 2016.
SIGNIFICANCE
I have mixed feelings about the significance of this paper. I found the experiments interesting, but I don’t feel that they reveal anything particularly surprising or unexpected about recurrent networks; it is hard to see how any of the experimental results will change the way either that I think about RNNs, or the way that I will use them in my own future work. On the other hand it is valuable to see intuitive results about RNNs confirmed by rigorous experiments, especially since few researchers have the computational resources to perform such large-scale experiments.
The capacity experiments (both per-parameter capacity and per-unit capacity) essentially force the network to model random data. For most applications of RNNs, however, we do not expect them to work with random data; instead when applied in machine translation or language modeling or image captioning or any number of real-world tasks, we hope that RNNs can learn to model data that is anything but random. It is not clear to me that an architecture’s ability to model random data should be beneficial in modeling real-world data; indeed, the experiments in Section 2.1 show that architectures vary in their capacity to model random data, but the text8 experiments in Section 3 show that these same architectures do not significantly vary in their capacity to model real-world data.
I do not think that the experimental results in the paper are sufficient to prove the significance of the proposed UGRNN and +RNN architectures. It is interesting that the UGRNN can achieve comparable bits per parameter as the ungated RNN and that the deep +RNNs are more easily trainable than other architectures, but the only experiments on a real-world task (language modeling on text8) do not show these architectures to be significantly better than GRU or LSTM.
SUMMARY
I wish that the experiments had revealed more surprising insights about RNNs, though there is certainly value in experimentally verifying intuitive results. The proposed UGRNN and +RNN architectures show some promising results on synthetic tasks, but I wish that they showed more convincing performance on real-world tasks. Overall I think that the good outweighs the bad, and that the ideas of this paper are of value to the community.
PROS
- The paper is the first of my knowledge to explicitly measure the bits per parameter that RNNs can store
- The paper experimentally confirms several intuitive ideas about RNNs:
    - RNNs of any architecture can store about one number per hidden unit from the input
    - Different RNN architectures should be compared by their parameter count, not their hidden unit count
    - With very careful hyperparameter tuning, all RNN architectures perform about the same on text8 language modeling
    - Gated architectures are easier to train than non-gated RNNs
CONS
- Experiments do not reveal anything particularly surprising or unexpected
- The UGRNN and +RNN architectures do not feel well-motivated
- The utility of the UGRNN and +RNN architectures is not well-established


DEEP REINFORCEMENT LEARNING
When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations. We also compare our learned experimentation policies to randomized baselines and show that the learned policies lead to better predictions.
1 INTRODUCTION
Our work is inspired by empirical findings and theories in psychology indicating that infant learning and thinking is similar to that of adult scientists (Gopnik, 2012). One important view in developmental science is that babies are endowed with a small number of separable systems of core knowledge for reasoning about objects, actions, number, space, and possibly social interactions (Spelke & Kinzler, 2007). The object core system covering aspects such as cohesion, continuity, and contact, enables babies and other animals to solve object related tasks such as reasoning about oclusion and predicting how objects behave.
Core knowledge research has motivated the development of methods that endow agents with physics priors and perception modules so as to infer intrinsic physical properties rapidly from data (Battaglia et al., 2013; Wu et al., 2015; 2016; Stewart & Ermon, 2016). For instance, using physics engines and mental simulation, it becomes possible to infer quantities such as mass from visual input (Hamrick et al., 2016; Wu et al., 2015).
In early stages of life, infants spend a lot of time interacting with objects in a seemingly random manner (Smith & Gasser, 2005). They interact with objects in multiple ways, including throwing, pushing, pulling, breaking, and biting. It is quite possible that this process of actively engaging with objects and watching the consequences of their actions helps infants understand different physical properties of the object which cannot be observed directly using their sensory systems. It seems infants run a series of “physical” experiments to enhance their knowledge about the world (Gopnik, 2012). The act of performing an experiment is useful both for quickly adapting an agent’s policy to a new environment and for understanding object properties in a holistic manner. Despite impressive advances in artificial intelligence that have led to superhuman performance in Go, Atari and natural language processing, it is still unclear if these systems behind these advances can rival the scientific intuition of even a small child.
While we draw inspiration from child development, it must be emphasized that our purpose is not to provide an account of learning and thinking in humans, but rather to explore how similar types of understanding might be learned by artificial agents in a grounded way. To this end we show that we can build agents that can learn to experiment so as to learn representations that are informative about physical properties of objects, using deep reinforcement learning. The act of conducting an experiment involves the agent having a belief about the world, which it then updates by observing the consequences of actions it performs.
We investigate the ability of agents to learn to perform experiments to infer object properties through two environments—Which is Heavier and Towers. In the Which is Heavier environment, the agent is able to apply forces to blocks and it must infer which of the blocks is the heaviest. In the Towers environment the agent’s task is to infer how many rigid bodies a tower is composed of by knocking it down. Unlike Wu et al. (2015), we assume that the agent has no prior knowledge about physical properties of objects, or the laws of physics, and hence must interact with the objects in order to learn to answer questions about these properties.
Our results indicate that in the case Which is Heavier environment our agents learn experimentation strategies that are similar to those we would expect from an algorithm designed with knowledge of the underlying structure of the environment. In the Towers environment we show that our agents learn a closed loop policy that can adapt to a varying time scale. In both environments we show that when using the learned interaction policies agents are more accurate and often take less time to produce correct answers than when following randomized interaction policies.
2 WHAT IS THIS PAPER ABOUT?
This is an unusual paper in that it does not present a new model or propose a new algorithm. There is a reinforcement learning task at the core of each of our experiments, but the algorithm and models we use to solve it are not new, and many other existing approaches should be expected to perform equally well if they were to be substituted in the same setting.
This paper is a step towards agents that understand objects and intuitive reasoning in physical worlds. Our best AI agents currently fail on simple control tasks and simple games, such as Montezuma’s Revenge, because when they look at a screen that has a ladder, a key and a skull they don’t immediately know that keys open doors, that skulls are probably hazardous and best avoided, that ladders allow us to defy gravity, etc. The understanding of physics, relations and objects enables children to solve seemingly simple problems that our best existing AI agents do not come close to begin to solve.
Endowing our agents with knowledge of objects would help enormously with planning, reasoning and exploration, and yet, doing so is far from trivial. What is an object? It turns out this question does not have a straightforward answer, and this paper is based around the idea that staring at a thing is not enough to understand what it is.
Children understand their world by engaging with it. Poking something to find that it is soft, tasting it to discover it is delicious, or hitting it to see if it falls down. Much of the knowledge people have of the world is the result of interaction. Vision or open loop perception alone is not enough.
This paper introduces tasks where we can evaluate the ability of agents to learn about these “hidden” properties of objects. This requires environments where the tasks depend on these properties (otherwise the agents have no incentive to learn about them) and also that we have a way to probe for this understanding in agents that complete the tasks.
Previous approaches to this problem have relied on either explicit knowledge of the underlying structure of the environment (e.g. hard-wired physical laws) or on exploiting correlations between material appearance and physical properties (see Section 7 for much more detail). One of the contributions of this paper is to show that our agents can still learn about properties of objects, even when the connection between material appearance and physical properties is broken. This setting allows us to show that our agents are not merely learning that blocks are heavy; they are learning how to check if blocks are heavy.
None of the previous approaches give a complete account of how agents could come to understand the physical properties of the world around them. Specifying a model manually is difficult to scale,
generalize and to ground in perception. Making predictions from only visual properties will fail to distinguish between objects that look similar, and it will certainly be unable to distinguish between a sack full of rocks and a sack full of tennis balls.
3 ANSWERING QUESTIONS THROUGH INTERACTION
We pose the problem of experimentation as that of answering questions about non-visual properties of objects present in the environment. We design environments that ask questions about these properties by providing rewards when the agent is able to infer them correctly, and we train agents to answer these questions using reinforcement learning.
We design environments that follow a three phase structure:
Interaction Initially there is an exploration phase, where the agent is free to interact with the environment and gather information.
Labeling The interaction phase ends when the agent produces a labeling action through which it communicates its answer to the implicit question posed by the environment.
Reward When the agent produces as labeling action, the environment responds with a reward, positive for a correct answer and negative for incorrect, and the episode terminates. The episode terminates automatically with a negative reward if the agent does not produce a labeling action before a maximum time limit is reached.
Crucially, the transition between interaction and labeling does not happen at a fixed time, but is initiated by the agent. This is achieved by providing the agent with the ability to produce either an interaction action or a labeling action at every time step. This allows the agent to decide when enough information has been gathered, and forces it to balance the trade-off between answering now given its current knowledge, or delaying its answer to gather more information.
The optimal trade-off between information gathering and risk of answering incorrectly depends on two factors. The first factor is the difficulty of the question and the second is the cost of information. The difficulty is environment specific and is addressed later when we describe the environments. The cost of information can be generically controlled by varying the discount factor during learning. A small discount factor places less emphasis on future rewards and encourages the agent to answer as quickly as possible. On the other hand, a large discount factor encourages the agent to spend more time gathering information in order to increase the likelihood of choosing the correct answer.
Our use of “questions” and “answers” differs from how these terms are used elsewhere in the literature. Sutton et al. (2011) talk about a value function as a question, and the agent provides an answer in the form of an approximation of the value. The answer incorporates the agent’s knowledge, and the match between the actual value and the agent’s approximation grounds what it means for this knowledge to be accurate.
In our usage the environment (or episode) itself is the question, and answers come in the form of labeling actions. In each episode there is a correct answer whose semantics is grounded in the sign of the reward function, and the accuracy of an agents knowledge is assessed by the frequency with which it is able to choose the correct answer.
Using reward (rather than value) to ground our semantics means that we have a straightforward way to ask questions that do not depend on the agent’s behavior. For example, we can easily ask the question “Which block is heaviest?” without making the question contingent on a particular information acquisition strategy.
4 AGENT ARCHITECTURE AND TRAINING
We use the same basic agent architecture and training procedure for all of our experiments, making only minimal modifications in order to adapt the agents to different observation spaces and actuators. For all experiments we train recurrent agents using an LSTM with 100 hidden units. When working from features we feed the observations into the LSTM directly. When training from pixels we first scale the observations to 84x84 pixels and feed them through a three convolution layers, each
followed by a ReLU non-linearity. The three layers have 32, 64, 64 square filters with sizes 8, 4, 3, which are applied at strides of 4, 2, 1 respectively. We train the agents using Asynchronous Advantage Actor Critic (Mnih et al., 2016), but ensure that the unroll length is always greater than the timeout length so the agent network is unrolled over the entirety of each episode.
5 WHICH IS HEAVIER
The Which is Heavier environment is designed to ask a question about the relative masses of different objects in a scene. We assign masses to objects in a way that is uncorrelated with their appearance in order to ensure that the task is not solvable without interaction.
5.1 ENVIRONMENT
The environment is diagrammed in the left panel of Figure 1. It consists of four blocks, which are constrained to only move vertically. The blocks are always the same size, but vary in mass between episodes. The agent’s strength (i.e. magnitude of force it can apply) remains constant between episodes.
The question to answer in this environment is which of the four blocks is the heaviest. Since the mass of each block is randomly assigned in each episode, the agent must poke the blocks and observe how they respond in order to make this determination. Assigning masses randomly ensures it is not possible to solve this task from vision (or features) alone, since the appearance and identity of each block imparts no information about its mass in the current episode. The only way to obtain information about the masses of the blocks is to interact with them and watch how they respond.
The Which is Heavier environment is designed to encode a latent bandit problem through a “physical” lens. Each block corresponds to an arm of the bandit, and the reward obtained by pulling each arm is proportional to the mass of the block. Identifying the heaviest block can then be seen as a best arm identification problem (Audibert & Bubeck, 2010). Best arm identification is a well studied problem in experimental design, and understanding of how an optimal solution to the latent bandit should behave is used to guide our analysis of the agents we train on this task.
It is important to emphasize that we cannot simply apply standard bandit algorithms here, because we impose a much higher level of prior ignorance on our algorithms than that setting allows. Bandit algorithms assume that rewards are observed directly, whereas our agents observe mass through its role in dynamics (and in the case of learning from pixels, through the lens of vision as well). To maintain a bandit setting one could imagine parameterizing this transformation from reward to observation, and perhaps even learning the mapping as well; however, doing so requires explicitly acknowledging the mapping in the design of the learning algorithm, which we avoid doing. Moreover, acknowledging this mapping in any way requires the a-priori recognition of the existence of the latent bandit structure. From the perspective of our learning algorithm the mere existence of such a structure also lies beyond the veil of ignorance.
Controlling the distribution of masses allows us to control the difficulty of this task. In particular, by controlling the size of the mass gap between the two heaviest blocks we can make the task more
or less difficult. We generate masses in the range [0, 1] and scale them to an appropriate range for the agent’s strength.
We use the following scheme for controlling the difficulty of the Which is Heavier environment. First we select one of the blocks uniformly at random to be the “heavy” block and designate the remaining three as “light” blocks. We sample the mass of the heavy block from Beta(β, 1) and the mass of the light blocks from Beta(1, β). The single parameter β effectively controls the distribution of mass gaps (and thus controls the difficulty), with large values of β leading to easier problems. Figure 1 shows the distribution of mass gaps for three values of β that we use in our experiments.
We distinguish between problem level and instance level difficulty for this domain. Instance level difficulty refers to the size of the mass gap in a single episode. If the mass gap is small it is harder to determine which block is heaviest, and we say that one episode is more difficult than another by comparing their mass gaps. Problem level difficulty refers to the shape of the generating distribution of mass gaps (e.g. as shown in the right panel of Figure 1). A distribution that puts more mass on configurations that have a small mass gap will tend to generate more episodes that are difficult at the instance level, and we say that one distribution is more difficult than another if it is more likely to generate instances with small mass gaps. We control the problem level difficulty through β, but we incorporate both problem and instance level difficulty in our analysis.
We set the episode length limit to 100 steps in this environment, which is sufficient time to be much longer than a typical episode by a successfully trained agent.
5.2 ACTUATORS
The obvious choice for actuation in physical domains is some kind of arm or hand based manipulator. However, controlling an arm or hand is quite challenging on its own, requiring a fair amount of dexterity on the part of the agent. The manipulation problem, while very interesting in its own right, is orthogonal to our goals in this work. Therefore we avoid the problem of learning dexterous manipulation by providing the agent with a much simpler form of actuation.
We call the actuation strategy for this environment direct actuation, which allows the agent to affect forces on the different blocks directly. At every time step the agent can output one out of eight possible actions. The first four actions result in an application of a vertical force of fixed magnitude to center of mass of each of the four blocks respectively. The remaining actions are labeling actions and correspond to agent’s selection of which is the heaviest block.
5.3 EXPERIMENTS
Our first experiment is a sanity check to show that we can train agents successfully on the Which is Heavier environment using both features and pixels. This experiment is designed simply to show that our task is solvable, and to illustrate that by changing the problem difficulty we can make the task very hard.
We present two additional experiments showing how varying difficulty leads to differentiated behavior both at the problem level and at the instance level. In both cases knowledge of the latent bandit problem allows us to make predictions about how an experimenting agent should behave, and our experiments are designed to show that qualitatively correct behavior is obtained by our agents in spite of their a-priori ignorance of the underlying bandit problem.
We show that as we increase the problem difficulty the learned policies transition from guessing immediately when a heavy block is found to strongly preferring to poke all blocks before making a decision. This corresponds to the observation that if it is unlikely for more than one arm to give high reward then any high reward arm is likely to be best.
We also observe that our agents can adapt their behavior to the difficulty of individual problem instances. We show that a single agent will tend to spend longer gathering information when the particular problem instance is more difficult. This corresponds to the observation that when the two best arms have similar reward then more information is required to accurately distinguish them.
Finally, we conduct an experiment comparing our learned information gathering policies to a randomized baseline method. This experiment shows that agents more reliably produce the correct label by following their learned interaction policies than by observing the environment being driven by random actions.
Success in learning For this experiment we trained several agents at three different difficulties corresponding to β ∈ {3, 5, 10}. For each problem difficulty we trained agents on both feature observations, which includes the z coordinate of each of the four blocks; and also using raw pixels, providing 84 × 84 pixel RGB rendering of the scene to the agent. Representative learning curves for each condition are shown in Figure 2. The curves are smoothed over time and show a running estimate of the probability of success, rather than showing the reward directly.
The agents do not reach perfect performance on this task, with more difficult problems plateauing at progressively lower performance. This can be explained by looking at the distributions of instance level difficulties generated by different settings of β, which is shown in the right panel of Figure 1. For higher difficulties (lower values of β) there is a substantial probability of generating problem instances where the mass gap is near 0, which makes distinguishing between the two heaviest blocks very difficult.
Population strategy differentiation For this experiment we trained agents at three different difficulties corresponding to β ∈ {3, 5, 10} all using a discount factor of γ = 0.95 which corresponds a relatively high cost of gathering information. We trained three agents for each difficulty and show results aggregated across the different replicas.
After training, each agent was run for 10,000 steps under the same conditions they were exposed to during training. We record the number and length of episodes executed during the testing period as well as the outcome of each episode. Episodes are terminated by timeout after 100 steps, but the vast majority of episodes are terminated in < 30 steps by the agent producing a label. Since episodes vary in length not all agents complete the same number of episodes during testing.
The left plot in Figure 3 shows histograms of the episode lengths broken down by task difficulty. The dashed vertical line indicates an episode length of four interaction steps, which is the minimum number of actions required for the agents to interact with every block. At a task difficulty of β = 10 the agents appear to learn simply to search for a single heavy block (which can be found with an average of two interactions). However, at a task difficulty of β = 3 we see a strong bias away from terminating the episode before taking at least four exploratory actions.
Individual strategy differentiation For this experiment we trained agents using the same three task difficulties as in the previous experiment, but with an increased discount factor of γ = 0.99.
This decreases the cost of exploration and encourages the agents to gather more information before producing a label, leading to longer episodes.
After training, each agent was run for 100,000 steps under the same conditions they were exposed to during training. We record the length of each episode, as well as the mass gap between the two heaviest blocks in each episode. In the same way that we use the distribution of mass gaps as a measure of task difficulty, we can use the mass gap in a single episode as a measure of the difficulty of that specific problem instance. We again exclude from analysis the very small proportion of episodes that terminate by timeout.
The right plots in Figure 3 show the relationship between the mass gap and episode length across the testing runs of two different agents. From these plots we can see how a single agent has learned to adapt its behavior based on the difficulty of a single problem instance. Although the variance is high, there is a clear correlation between the mass gap and the length of the episodes. This behavior reflects what we would expect from a solution to the latent bandit problem; more information is required to identify the best arm when the second best arm is nearly as good.
Randomized interaction For this experiment we trained several agents using both feature and pixel observations at the same three task difficulties with a discount of γ = 0.95. In total we trained six sets of agents for this experiment.
After training, each agent was run for 10,000 steps under the same conditions used during training. We record the outcome of each episode, as well as the number of steps taken by each agent before it chooses a label. For each agent we repeat the experiment using both the agent’s learned interaction policy as well as a randomized interaction policy.
The randomized interaction policy is obtained as follows: At each step the agent chooses a candidate action using its learned policy. If the candidate action is a labeling action then it is passed to the environment unchanged (and the episode terminates). However, if the candidate action is an interaction action then we replace the agent action with a new interaction action chosen uniformly at random from the available action set. When following the randomized interaction policy the agent has no control over the information gathering process, but still controls when each episode ends, and what label is chosen.
Figure 4 compares the learned interaction policies to the randomized interaction baselines. The results show that the effect on episode length is small, with no consistent bias towards longer or shorter episodes across difficulties and observation types. However, the learned interaction policies produce more accurate labels across all permutations.
6 TOWERS
The Towers environment is designed to ask agents to count the number of cohesive rigid bodies in a scene. The environment is designed so that in its initial configuration it is not possible to determine the number of rigid bodies from vision or features alone.
6.1 ENVIRONMENT
The environment is diagrammed in the left panel of Figure 5. It consists of a tower of five blocks which can move freely in three dimensions. The initial block tower is always in the same configuration but in each episode we bolt together different subsets of the blocks to form larger rigid bodies as shown in the figure.
The question to answer in this environment is how many rigid bodies are formed from the primitive blocks. Since which blocks are bound together is randomly assigned in each episode, and binding forces are invisible, the agent must poke the tower and observe how it falls down in order to determine how many rigid bodies it is composed of. We parameterize the environment in such a way that the distribution over the number of separate blocks in the tower is uniform. This ensures that there is no single action strategy that achieves high reward.
6.2 ACTUATORS
In the Towers environment, we used two actuators: direct actuation, which is similar to the Which is Heavier environment; and the fist actuator, described below. In case of the direct actuation, the agent can output one out of 25 actions. At every time step, the agent can apply a force of fixed magnitude in either of +x, -x, +y or -y direction to one out of the five blocks. If two blocks are glued together, both blocks move under the effect of force. We use towers of five blocks, which results in 20 different possible actions. The remaining actions are labeling actions that are used by the agent to indicate the number of distinct blocks in the tower.
The fist is a large spherical object that the agent can actuate by setting velocities in a 2D horizontal plane. Unlike direct actuation, the agent cannot apply any direct forces to the objects that constitute the tower, but only manipulate them by pushing or hitting them with the fist. At every time step agent can output one of nine actions. The first four actions corresponds to setting the velocity of the fist to a constant amount in (+x, -x, +y, -y) directions respectively. The remaining actions are labeling actions, that are used by the agent to indicate the number of distinct blocks in the tower.
In order to investigate if the agent learns a strategy of stopping after a fixed number of time steps or whether it integrates sensory information in a non-trivial manner we used a notion of “control time step”. The idea of control time step is similar to that of action repeats and if the physics simulation time step is 0.025s and control time step is 0.1s, it means that the same action is repeated 4 times. For the direct actuators we use an episode timeout of 26 steps and for both actuator types.
6.3 EXPERIMENTS
Our first experiment is again intended to show that we can train agents in this environment. We show simply that the task is solvable by our agents using both types of actuation.
The second experiment shows that the agents learn to wait for an observation where they can identify the number of rigid bodies before producing an answer. This is designed to show that the agents find a closed loop strategy for counting the number of rigid bodies. An alternative hypothesis would be that agents learn to wait for (approximately) the same number of steps each time and then take their best guess.
Our third experiment compares the learned policy to a randomized interaction policy and shows that agents are able to determine the correct number of blocks in the tower more quickly and more reliably when using their learned policy to gather information.
Success in learning For this experiment we trained several agents on the Towers environment using different pairings of actuators and perception. The features observations include the 3d position of each primitive block, and when training using raw pixels we provide an 84× 84 pixel RGB rendering of the scene as the agent observation. Figure 6 shows learning curves for each combination of actuator and observation type.
In all cases we obtain agents that solve the task nearly perfectly, although when training from pixels we find that the range of hyperparameters which train successfully is narrower than when training from features. Interestingly, the fist actuators lead to the fastest learning, in spite of the fact that the agent must manipulate the blocks indirectly through the fist. One possible explanation is that the fist can affect multiple blocks in one action step, whereas in the direct actuation only one block can be affected per time step.
Waiting for information For this experiment we trained an agent with pixel observations and the fist actuator on the towers task with an control time step of 0.1 seconds and examine its behavior at test time with a smaller delay between actions. Reducing the control time step means that from the agent perspective time has been slowed down. Moving the fist a fixed amount of distance takes longer, as does waiting for the block tower to collapse once it has been hit.
After training the agent was run for 10000 steps for a range of different control time steps. We record the outcome of each episode, as well as the number of steps taken by the agent before it chooses a label. None of the test episodes terminate by timeout, so we include all of them in the analysis.
The plot in Figure 5 shows the probability of answering correctly, as well as the median length of each episode measured in seconds. In terms of absolute performance we see a small drop compared to the training setting, where the agent is essentially perfect, but the agent performance remains good even for substantially smaller control timesteps than were used during training.
We also observe that the episodes with different time steps take approximate the same amount of real time across the majority of the tested range. This corresponds to a large change in episode length as measured by number of agent actions, since with an control time step of 0.01 the agent must execute 10x as many actions to cover the same amount of real time as compared to the control time step used during training. From this we can infer that the agent has learned to wait for an informative observation before producing a label, as opposed to a simpler degenerate strategy of waiting a fixed amount of steps before answering.
Randomized interaction For this experiment we trained several agents for each combination of actuator and observation type, and examine their behavior when observing an environment driven by a random interaction policy. The randomized interaction policy is identical to the randomized baseline used in the Which is Heavier environment.
After training, each agent was run for 10,000 steps. We record the outcome of each episode, as well as the number of steps taken by the agent before it chooses a label. For each agent we repeat the experiment using both the agent’s learned interaction policy as well as the randomized interaction policy.
Figure 7 compares the learned interaction policies to the randomized interaction baselines. The results show that the agents tend to produce labels more quickly when following their learned interaction policies, and also that the labels they produce in this way are much more accurate.
7 RELATED WORK
Deep learning techniques in conjunction with vast labeled datasets have yielded powerful models for image classification (Krizhevsky et al., 2012; He et al., 2016) and speech recognition (Hinton et al., 2012). In recent years, as we have approached human level performance on these tasks, there has been a strong interest in the computer vision field in moving beyond semantic classification, to tasks that require a deeper and more nuanced understanding of the world.
Inspired by developmental studies (Smith & Gasser, 2005), some recent works have focused on learning representations by predicting physical embodiment quantities such as ego-motion (Agrawal et al., 2015; Jayaraman & Grauman, 2015), instead of symbolic labels. Extending the realm of things-to-be-predicted to include quantities beyond class labels, such as viewer centric parameters (Doersch et al., 2015) or the poses of humans within a scene (Delaitre et al., 2012; Fouhey et al., 2014), has been shown to improve the quality of feature learning and scene understanding. Researchers have looked at cross modal learning, for example synthesizing sounds from visual images (Owens et al., 2015), using summary statistics of audio to learn features for object recognition (Owens et al., 2016) or image colorization (Zhang et al., 2016).
Inverting the prediction tower, another line of work has focused on learning about the visual world by synthesizing, rather than analyzing, images. Major cornerstones of recent work in this area include the Variational Autoencoders of Kingma & Welling (2014), the Generative Adversarial Networks
of (Goodfellow et al., 2014), and more recently autoregressive models have been very successful (van den Oord et al., 2016).
Building on models of single image synthesis there have been many works on predicting the evolution of video frames over time (Ranzato et al., 2014; Srivastava et al., 2015; van den Oord et al., 2016). Xue et al. (2016) have approached this problem by designing a variational autoencoder architecture that uses the latent stochastic units of the VAE to make choices about the direction of motion of objects, and generates future frames conditioned on these choices.
A different form of uncertainty in video prediction can arise from the effect of actions taken by an agent. In environments with deterministic dynamics (where the possibility of “known unknowns” can, in principle, be eliminated), very accurate action-conditional predictions of future frames can be made (Oh et al., 2015). Introducing actions into the prediction process amounts to learning a latent forward dynamics model, which can be exploited to plan actions to achieve novel goals (Watter et al., 2015; Assael et al., 2015; Fragkiadaki et al., 2016). In these works, frame synthesis plays the role of a regularizer, preventing collapse of the feature space where the dynamics model lives.
Agrawal et al. (2016) break the dependency between frame synthesis and dynamics learning by replacing frame synthesis with an inverse dynamics model. The forward model plays the same role as in the earlier works, but here feature space collapse is prevented by ensuring that the model can decode actions from pairs of time-adjacent images. Several works, including Agrawal et al. (2016) and Assael et al. (2015) mentioned above but also Pinto et al. (2016); Pinto & Gupta (2016); Levine et al. (2016), have gone further in coupling feature learning and dynamics. The learned dynamics models can be used for control not only after learning but also during the learning process in order to collect data in a more targeted way, which has been shown to improve the speed and quality of learning in robot manipulation tasks.
A key challenge of learning from dynamics is collecting the appropriate data. An ingenious solution to this is to import real world data into a physics engine and simulate the application of forces in order to generate ground truth data. This is the approach taken by Mottaghi et al. (2016), who generate an “interactable” data set of scenes, which they use to generate a static data set of image and force pairs, along with the ground truth trajectory of a target object in response to the application of the indicated force.
When the purpose is learning an intuitive understanding of dynamics it is possible to do interesting work with entirely synthetic data (Fragkiadaki et al., 2016; Lerer et al., 2016). Lerer et al. (2016) show that convolutional networks can learn to make judgments about the stability of synthetic block towers based on a single image of the tower. They also show that their model trained on synthetic data is able to generalize to make accurate judgments about photographs of similar block towers built in the real world.
Making intuitive judgments about block towers has been extensively studied in the psychophysics literature. There is substantial evidence connecting the behavior of human judgments to inference over an explicit latent physics model (Hegarty, 2004; Hamrick et al., 2011; Battaglia et al., 2013). Humans can infer mass by watching movies of complex rigid body dynamics (Hamrick et al., 2016).
A major component of the above line of work is analysis by synthesis, in which understanding of a physical process is obtained by learning to invert it. Observations are assumed to be generated from an explicitly parameterized generative model of the true physical process, and provide constraints to an inference process run over the parameters of this model. The analysis by synthesis approach has been extremely influential due to its power to explain human judgments and generalization patterns in a variety of situations (Lake et al., 2015).
Galileo (Wu et al., 2015) is a particularly relevant instance of tying together analysis by synthesis and deep learning for understanding dynamics. This system first infers the physical parameters (mass and friction coefficient) of a variety of blocks by watching videos of them sliding down slopes and colliding with other blocks. This stage of the system uses an off-the-shelf object tracker to ground inference over the parameters of a physical simulator, and the inference is achieved by matching simulated and observed block trajectories. The inferred physical parameters are used to train a deep network to predict the physical parameters from the initial frame of video. At test time the system is evaluated by using the deep network to infer physical parameters of new blocks, which can be fed into the physics engine and used to answer questions about behaviors not observed at training time.
Physics 101 (Wu et al., 2016) is an extension of Galileo that more fully embraces deep learning. Instead of using a first pass of analysis by synthesis to infer physical parameters based on observations, a deep network is trained to regress the output of an object tracker directly, and the relevant physical laws are encoded directly into the architecture of the model. The authors show that they can use latent intrinsic physical properties inferred in this way to make novel predictions. The approach of encoding physical models as architecture constraints has also been proposed by Stewart & Ermon (2016).
Many of the works discussed thus far, including Galileo and Physics 101, are restricted to passive sensing. Pinto et al. (2016); Pinto & Gupta (2016); Agrawal et al. (2016); Levine et al. (2016) are exceptions to this because they learn their models using a sequential greedy data collection bootstrapping strategy. Active sensing, it appears, is an important aspect of visual object learning in toddlers as argued by Bambach et al. (2016), providing motivation for the approach presented here.
In computer vision, it is well known that recognition performance can be improved by moving so as to acquire new views of an object or scene. Jayaraman & Grauman (2016), for example, apply deep reinforcement learning to construct an agent that chooses how to acquire new views of an object so as to classify it into a semantic category, and their related work section surveys many other efforts in active vision.
While Jayaraman & Grauman (2016) and others share deep reinforcement learning and active sensing in common with our work, their goal is to learn a policy that can be applied to images to make decisions based on vision. In contrast, the goal in this paper is to study how agents learn to experiment continually so as to learn representations to answer questions about intrinsic properties of objects. In particular, our focus is on tasks that can only be solved by interaction and not by vision alone.
8 CONCLUSION AND FUTURE DIRECTIONS
Despite recent advances in artificial intelligence, machines still lack a common sense understanding of our physical world. There has been impressive progress in recognizing objects, segmenting object boundaries and even describing visual scenes with natural language. However, these tasks are not enough for machines to infer physical properties of objects such as mass, friction or deformability.
We introduce a deep reinforcement learning agent that actively interacts with physical objects to infer their hidden properties. Our approach is inspired by findings from the developmental psychology literature indicating that infants spend a lot of their early time experimenting with objects through random exploration (Smith & Gasser, 2005; Gopnik, 2012; Spelke & Kinzler, 2007). By letting our agents conduct physical experiments in an interactive simulated environment, they learn to manip-
ulate objects and observe the consequences to infer hidden object properties. We demonstrate the efficacy of our approach on two important physical understanding tasks—inferring mass and counting the number of objects under strong visual ambiguities. Our empirical findings suggest that our agents learn different strategies for these tasks that balance the cost of gathering information against the cost of making mistakes in different situations.
Scientists and children are able not only to probe the environment to discover things about it, but they can also leverage their findings to answer new questions. In this paper we have shown that agents can be trained to gather knowledge to answer questions about hidden properties, but we have not addressed the larger issue of theory building, or transfer of this information. Given agents that can make judgments about mass and numerosity, how can they be enticed to leverage this knowledge to solve new tasks?
Another important aspect of understanding through interaction is that that the shape of the interactions influences behavior. We touched on this in the Towers environment where we looked at two different actuation styles, but there is much more to be done here. Thinking along these lines leads naturally to exploring tool use. We showed that agents can make judgments about object mass by hitting them, but could we train an agent to make similar judgments using a scale?
Finally, we have made no attempt in this work to optimize data efficiency, but learning physical properties from fewer samples is an important direction to pursue.
ACKNOWLEDGMENTS
We would like to thank Matt Hoffman for several enlightening discussions about bandits. We would also like to thank the ICLR reviewers, whose helpful feedback allowed us to greatly improve the paper.

This paper purports to investigate the ability of RL agents to perform ‘physics experiments’ in an environment, to infer physical properties about the objects in that environment. The problem is very well motivated; indeed, inferring the physical properties of objects is a crucial skill for intelligent agents, and there has been relatively little work in this direction, particularly in deep RL. The paper is also well-written.
As there are no architectural or theoretical contributions of the paper (and none are claimed), the main novelty comes in the task application – using a recurrent A3C model for two tasks that simulate an agent interacting with an environment to infer physical properties of objects. More specifically, two tasks are considered – moving blocks to determine their mass, and poking towers such that they fall to determine the number of rigid bodies they are composed of. These of course represent a very limited cross-section of the prerequisite abilities for an agent to understand physics. This in itself is not a bad thing, but since there is no comparison of different (simpler) RL agents on the tasks, it is difficult to determine if the tasks selected are challenging. As mentioned in the pre-review question, the ‘Which is Heavier’ task seems quite easy due to the actuator set-up, and the fact that the model simply must learn to take the difference between successive block positions (which are directly encoded as features in most experiments).  Thus, it is not particularly surprising that the RL agent can solve the proposed tasks.
The main claim beyond solving two proposed tasks related to physics simulation is that “the agents learn different strategies for these tasks that balance the cost of gathering information against the cost of making mistakes”. The ‘cost of gathering information’ is implemented by multiplying the reward with a value of gamma < 1. This is somewhat interesting behaviour, but is hardly surprising given the problem setup.
One item the authors highlight is that their approach of learning about physical object properties through interaction is different from many previous approaches, which use visual cues. However, the authors also note that this in itself is not novel, and has been explored in other work (e.g. Agrawal et al. (2016)). I think it’s crucial for the authors to discuss these approaches in more detail (potentially along with removing some other, less relevant information from the related work section), and specifically highlight why the proposed tasks in this paper are interesting compared to, for example, learning to move objects towards certain end positions by poking them.
To discern the level of contribution of the paper, one must ask the following questions:
1)	how much do these two tasks contribute (above previous work) to the goal of having agents learn the properties of objects by interaction; and
2)	how much do the results of the RL agent on these tasks contribute to our understanding of agents that interact with their environment to learn physical properties of objects?
It is difficult to know exactly, but due to the concerns outlined above, I am not convinced that the answers to (1) or (2) are “to a significant extent”. In particular, for (1), since the proposed agent is able to essentially solve both tasks, it is not clear that the tasks can be used to benchmark more advanced agents (e.g. it can’t be used as a set of bAbI-like tasks).
Another possible concern, as pointed out by Reviewer 3, is that the description of the model is extremely concise. It would be nice to have, for example, a diagram illustrating the inputs and outputs to the model at each time step, to ease replication.
Overall, it is important to make progress towards agents that can learn to discover physical properties of their environment, and the paper contributes in this direction. However, the technical contributions of this paper are rather limited – thus, it is not clear to what extent the paper pushes forward research in this direction beyond previous work that is mentioned. It would be nice, for example, to have some discussion about the future of agents that learn physics from interaction (speculation on more difficult versions of the tasks in this paper), and how the proposed approach fits into that picture.
---------------
EDIT: score updated, see comments below