Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time. One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers. We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient. Finally, we test our method on various networks and datasets. These experiments exhibit high pruning rates while maintaining the accuracy of the original network.
Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time. One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers. We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient. Finally, we test our method on various networks and datasets. These experiments exhibit high pruning rates while maintaining the accuracy of the original network.
1 INTRODUCTION
Neural networks and deep learning recently achieved state-of-the-art solutions to many problems in computer vision (Krizhevsky et al. (2012); He et al. (2015)), speech recognition (Graves et al. (2013)), natural language processing (Mikolov et al. (2013)) and reinforcement learning (Silver et al. (2016)). Using large and oversized networks in these tasks is a common practice. Such oversized networks can easily overfit on the training dataset while having poor generalization on the testing data (Sabo & Yu (2008)). A rule of thumb for obtaining useful generalization is to use the smallest number of parameters that can fit the training data (Reed (1993)). Unfortunately, this optimal size is not usually obvious and therefore the size of the neural networks is determined by a few rules-of-thumb (Heaton (2008)) which do not guarantee an optimal size for a given problem. One common approach to overcome overfitting is to choose an over-sized network and then apply regularization (Ng (2004)) and Dropout (Srivastava et al. (2014)). However, these techniques do not reduce the number of parameters and therefore do not resolve the high demand of resources at test time.
Another method is to start with an oversized network and then use pruning algorithms to remove redundant parameters while maintaining the network’s accuracy (Augasta & Kathirvalavakumar (2013)). These methods need to estimate the upper-bound size of a network, a task for which there are adequate estimation methods (Xing & Hu (2009)). If the size of a neural network is bigger than what is necessary, in theory, it should be possible to remove some of the extra neurons without affecting its accuracy. To achieve this goal, the pruning algorithm should find neurons which once removed result into no additional prediction errors. However, this may not be as easy as it sounds since all the neurons contribute to the final prediction and removing them usually leads to error.
It is easy to demonstrate this problem by fitting an oversized network on a toy dataset. Figure 1 shows a two-dimensional toy dataset which contains two linearly separable classes. Hence, only one hidden neuron in a two-layer perceptron should be enough to classify this data and any network with more than one neuron (such as the network in Figure 2-a) is an oversized network and can be pruned. However, there is no guarantee that removing one of the hidden neurons will maintain the network’s performance. As shown in the example in Figure 1, removing any of the hidden neurons results into a more compact, but under-performing network. Therefore, a more complicated process is required for pruning neural networks without accuracy loss.
Our goal in this paper is two-fold. First, we introduce NoiseOut, a pruning method based on the correlation between activations of the neurons. Second, we propose an approach which enforces the higher correlation between activations of neurons. Since the effectiveness of NoiseOut hinges on high correlations between neuron activations, the combination of these two methods facilitates more aggressive pruning.
2 RELATED WORK
Optimal Brain Damage (LeCun et al. (1989)) and Optimal Brain Surgeon (Hassibi & Stork (1993)) prune networks based on the Hessian of the loss function. It is shown that such pruning is more effective and more accurate than earlier magnitude-based pruning such as weight decay (Hanson & Pratt (1989)). However, the necessary second-order derivatives require additional computational resources.
Recently, replacing the fully connected layers with other types of layers has been utilized to reduced the number of parameters in a neural network. Deep fried convnets (Yang et al. (2015)) replaces these layers with kernel methods while the GoogLenet (Szegedy et al. (2015)) and Network in Network architecture (Lin et al. (2013)) replace them with global average pooling. Alternatively, Han et al. (2015) proposed a pruning method which learns the import connections between neurons, pruning the unimportant connections, and then retraining the remaining sparse network.
Besides pruning, other approaches have been proposed to reduce the computation and memory requirements of neural networks. HashNets( Chen et al. (2015)) reduce the storage requirement of neural networks by randomly grouping weights into hash buckets. These techniques can be combined with pruning algorithms to achieve even better performance. As an example, Deep Compression( Han et al. (2016)) proposed a three stage pipeline: pruning, trained quantization, and Huffman coding, to reduce the storage requirement of neural networks.
3 PROPOSED METHOD
In this section, we describe the details of the proposed method called NoiseOut. First, we show how this method can prune a single neuron and then how it can prune a full network, one neuron at a time.
3.1 PRUNING A SINGLE NEURON
The key idea in NoiseOut is to remove one of the two neurons with strongly correlated activations. The main rationale behind this pruning is to keep the signals inside the network as close to the original network as possible. To demonstrate this, assume there exists u, v, l such that |ρ(h(l)u , h(l)v ) ∣∣ = 1⇒ h (l) u = αh (l) v + β, where h (l) i is the activation of i th neuron in the lth layer. By definition:
h (l+1) k := ∑ i h (l) i w (l) i,k + b (l+1) k = ∑ i6=u,v h (l) i w (l) i,k + h (l) u w (l) u,k + h (l) v w (l) v,k + b (l+1) k
= ∑ i 6=u,v h (l) i w (l) i,k + αh (l) v w (l) u,k + βw (l) u,k + h (l) v w (l) v,k + b (l+1) k
= ∑ i 6=u,v h (l) i w (l) i,k + h (l) v ( αw (l) u,k + wv,k ) + ( βw (l) u,k + b (l+1) k ) (1)
This means that neuron u can be removed without affecting any of the neurons in the next layer, simply by adjusting the weights of v and neurons’ biases. Note that max
i,j,i 6=j ∣∣ρ(h(l)i , h(l)j )∣∣ = 1 is an ideal case. In this ideal scenario, removing one of the neurons results into no change in accuracy since the final output of the network will stay the same. In non-ideal cases, when the highest correlated neurons are not strongly correlated, merging them into one neuron may alter the accuracy. However, continuing the training after the merge may compensate for this loss. If this does not happen, it means that the removed neuron was necessary to achieve the target accuracy and the algorithm cannot compress the network any further without accuracy degradation.
NoiseOut follows the same logic to prune a single neuron using the following steps:
1. For each i, j, l calculate ρ ( h (l) i , h (l) j ) 2. Find u, v, l = argmax
u,v,l,u6=v ∣∣ρ(h(l)u , h(l)v )∣∣ 3. Calculate α, β := argmin
α,β
( h (l) u − αh(l)v − β ) 4. Remove neuron u in layer l 5. For each neuron k in layer l + 1:
- Update the weight w(l)v,k = w (l) v,k + αw (l) u,k
- Update the bias b(l+1)k = b (l+1) k + βw (l) u
3.2 ENCOURAGING CORRELATION BETWEEN NEURONS
The key element for successful pruning of neural networks using NoiseOut is the strong correlation between activation of the neurons. Essentially, a higher correlation between these activations means more efficient pruning. However, there is no guarantee that back-propagation results in correlated activations in a hidden layer. In this section, we propose a method to encourage higher correlation by adding additional output nodes, called noise outputs. The targets for noise outputs will randomly change in each iteration based on a predefined random distribution. We show that adding noise outputs to the output layer intensifies the correlation between activation in the hidden layers which will subsequently make the pruning task more effective.
3.2.1 EFFECT OF ADDING NOISE OUTPUTS TO THE NETWORK
To demonstrate the effect of adding noise outputs to the network, let us reconsider the toy example described previously in Figure 1, this time with some additional noise outputs in the output layer as shown in Figure 2-b. The result of training this network has been shown in Figure 3. As seen in this figure, the activation of the two hidden neurons has become highly correlated, and each hidden unit converged to the optimal discriminant by itself. This means that either of the extra neurons in the hidden layer can be removed without loss of accuracy.
This claim can be proved formally as well. The key to this proof is that neural networks are deterministic at inference time. In other words, the network generates a constant output for a constant input. For noise output, since the target is independent of the input, the training algorithm finds an optimal constant value that minimizes the expected error for every input. This objective can be presented as:
min W
C ( f(X;W ), [ Y, Ŷ ]) = min
W
( C ( fm(X;W ), Y ) + m∑ i=0 C ( f̂i(X;W ), Ŷi )) (2)
where W is the adjustable parameters (i.e. weights), X is the input, Y is the target, Ŷi is the target of noisy outputs at iteration i, C is the cost function, m is the number of iterations. fi(X;W ) and f̂i(X;W ) are the outputs of the neural network in the original and noise outputs respectively, at iteration i. Note that Ŷi changes in each iteration according to a random distribution PŶ .
The first part of Equation 2 represents the common objective of training a neural network, while the second part has been added because of the noise outputs. It is possible to adjust the effect of noise outputs based on PŶ . For instance by adding more noise outputs (in the case of Binomial distribution) or adjusting the variance (for Gaussian distribution). Another way of this adjustment is introducing a new multiplier for the second part of the cost. Although Ŷi changes in each iteration, the constant value that the network infers for any given input would be the same e.g. θ, due to the independent of PŶ from X . Therefore:
f̂m(X;W ) = J1×nθ
where J1×n is the matrix of ones of size 1×n (n is the number of samples in the dataset). The actual value of θ can be estimated for any given network architecture. As an example, let W (1), W (2) and Ŵ (2) be the weights of the first hidden layer, the output layer and the noisy neurons in a 2-2-1 MLP network respectively (Figure 2-b). Assuming linear activation in the hidden neurons and mean square error (MSE) as the cost function, Equation 2 can be rewritten as:
min W C = min W
C ( f(X;W ), [ Y, Ŷi ]) = min W (1),W (2) 1 2 (f(X;W )− Y )2 + min
W (1),Ŵ (2) m∑ i=0 1 2 ( f̂i(X;W )− Ŷi )2 = min W (1),W (2) ( f(X;W )− Y )2 + min W (1),Ŵ (2) 1 2 ( f̂(X;W )− θ )2 (3)
In this particular case, θ can be calculated using derivatives of the cost functions in Equation 3:
∂C ∂θ = ∂ ∂θ
[( f(X;W )− Y )2 + 1
2
( f̂(X;W )− θ )2] = ∂
∂θ [1 2 ( f̂(X;W )− θ)2 ] = ( θ − f̂(X;W ) ) = 0
⇒ θ = E [ f̂(X;W ) ] = E [ PŶ ]
(4)
This means that in this particular network with MSE as the cost function, the final error will be minimized when the network outputs the expected value of targets in noise outputs ( E [ PŶ ])
for any given input.
To demonstrate how outputting a constant value affects the weights of a network, let’s consider the network in Figure 2-b. In this case, the output of noisy output will be:
f̂(X;W ) = XW (1)Ŵ (2) = h (1) 1 w (2) 1,1 + h (1) 2 w (2) 2,1 = θ
⇒ h(1)1 = 1
w (2) 1,1
(θ − h(1)2 w2,1)
⇒ ρ h (1) 1 ,h (1) 2 = sgn (−w(2)2,1 w
(2) 1,1
) ⇒ ∣∣ρ h (1) 1 ,h (1) 2 ∣∣ = 1 (5) Equation 6 means that the activation of the hidden neurons has a correlation of 1 or -1. For more than two neurons it can be shown that the output of one neuron will be a linear combination of other neurons which means the claim still holds. The same results can be achieved empirically. Since the output of the noise outputs will converge to E [ PŶ ] , it seems that there may not be any difference between different random distributions with
the same expected value. Therefore, we tested different random distributions for E [ PŶ ]
with the same E [ PYN ] , on the network shown in Figure 2-b. These noise distributions are as follows:
Algorithm 1 NoiseOut for pruning hidden layers in neural networks 1: procedure TRAIN(X,Y ) . X is input, Y is expected output 2: W ← initialize_weights() 3: for each iteration do 4: YN ← generate_random_noise() . generate random expected values 5: Y ′ ← concatenate(Y, YN ) 6: W ← back_prop(X,Y ′) 7: while cost(W ) ≤ threshold do 8: A,B ← find_most_correlated_neurons(W,X) 9: α, β ← estimate_parameters(W,X,A,B) 10: W ′ ← remove_neuron(W,A) 11: W ′ ← adjust_weights(W ′, B, α, β) 12: W ←W ′ 13: return W
- Gaussian PŶ (x) = N (0.1, 0.4) Normal distribution with mean of 0.1 and standard deviation of 0.4. This noise distribution is appropriate for regression tasks with MSE cost.
- Binomial PŶ (x) = B(1, 0.1) Binomial distribution with 1 trial and success probability of 0.1. We chose binomial distribution since it generates random classification labels and is appropriate for networks that have to produce binary labels.
- Constant PŶ (x) = δ(x− 0.1) In this case, the target of the noise outputs is the constant value of 0.1. This is used as an expected-value “shortcut" so that we can examine a stochastic vs. a deterministic approach.
- No_Noise no noise output for comparison.
As it can be seen in the top row Figure 4, in a regular network with no noise unit (shown as No_Noise), the correlation between the output of hidden neurons ∣∣ρ h (1) 1 ,h (1) 2 ∣∣ does not go higher than 0.8, while in the existence of a noise output this value approaches to one, rather quickly. This means that the two hidden neuron are outputting just a different scale of the same value for any given input. In this case, NoiseOut easily prunes one of the two neurons.
The same technique can be applied to correlate the activation of the hidden neurons in networks with more than two layers. The bottom row of Figure 4 shows the correlation between the activation of two hidden neurons in the first layer of a six layer MLP (2-2-2-2-2-2-1). As it can be seen in this figure, adding noise outputs helped the neurons to achieve higher correlation compared to a network with no noise output. Binomial noise acts chaotic at the beginning due to its sudden change of expected values in the noise outputs while Gaussian noise improved the correlation the best in these experiments.
3.3 PRUNING A FULLY CONNECTED NETWORK
Algorithm 1 shows the final NoiseOut algorithm. For the sake of readability, this algorithm has been shown for networks with only one hidden layer. But the same algorithm can be applied to networks with more that one hidden layer by performing the same pruning on all the hidden layers independently. It can also be applied to convolutional neural networks that use dense layers, in which we often see over 90% of the network parameters (Cheng et al. (2015)).
This algorithm is simply repeating the process of removing a single neuron, as described in the previous section. The pruning ends when the accuracy of the network drops below some given threshold. Note that the pruning process is happening while training.
4 EXPERIMENTS
To illustrate the generality of our method we test it on a core set of common network architectures, including fully connected networks and convolutional neural networks with dense layers. In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold
for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network. For each experiment, different random distributions have been used for PŶ , to demonstrate the difference in practice.
4.1 PRUNING FULLY CONNECTED AND CONVOLUTIONAL NETWORKS
Table 1 and Table 2 show the result of pruning Lenet-300-100 and Lenet-5 (LeCun et al. (1998)) on MNIST dataset. Lenet-300-100 is a fully connected network with two hidden layers, with 300 and 100 neurons each, while Lenet-5 is a convolutional network with two convolutional layers and one dense layer. These networks achieve 3.05% and 0.95% error rate on MNIST respectively (LeCun et al. (1998)). Note that in Lenet-5 over 98% of parameters are in the dense layer and pruning them can decrease the model size significantly.
As it can be seen in these tables, NoiseOut removed over %95 of parameters with no accuracy degradation. Astonishingly, pruned Lenet-5 can achieve 0.95% error rate with only 3 neurons in the hidden layer which reduce the total number of weights in Lenet-5 by a factor of 44. Figure 6 demonstrates the output of these 3 neurons. This graph has been generated by outputting the activation of hidden layer neurons for 1000 examples randomly selected from MNIST dataset. Then, the data has been sorted by the target class. As it can be seen in this figure, the three neurons in the hidden layer efficiently encoded the output of convolutional layers to the expected ten classes. Obviously, these values can be utilized by the softmax layer to perform the final classification.
To test the effect of pruning of deeper architectures, we prune the network described in Table 4 on SVHN data set.This model which has over 1 million parameters, achieves 93.39% and 93.84% accuracy on training set and test set respectively. As it can be seen in Table 3, NoiseOut pruned more than 85% of the parameters from the base model while maintaining the accuracy.
Table 3: Pruning the reference network in Table 4, on SVHN dataset.
Method Dense Layer
Neurons Parameters Removed Parameters
Ground Truth 1024 1236250 -
No_Noise 132 313030 74.67% Gaussian 4 180550 85.39% Constant 25 202285 83.63% Bionomial 17 194005 84.30%
Table 4: Base model architecture for SVHN with 1236250 parameters.
Layer Filters Kernel Weights conv1 32 3× 3 896 conv2 32 3× 3 9246 conv3 32 3× 3 9246 pool1 - 2× 2 - conv4 48 3× 3 13872 conv5 48 3× 3 20784 conv6 48 3× 3 20784 pool2 - 2× 2 - conv7 64 3× 3 27712 conv8 64 3× 3 36928 conv9 64 3× 3 36928 pool3 - 2× 2 - dense 1024 - 1049600 softmax 10 - 10250
4.2 EFFECT OF NOISEOUT ON TEST ACCURACY
To explore the effect of NoiseOut on the test accuracy, we pruned Lenet-300-100 and Lenet-5 on MNIST with multiple accuracy thresholds, using Gaussian distribution as the target for noise outputs. In each one of these experiments, we measured both the training and test accuracy. As expected, the results which have been shown in Figure 5, indicate that lower accuracy thresholds result into more pruned parameters. However, the gap between training and testing threshold stays the same. This shows that pruning the network using NoiseOut does not lead to overfitting.
4.3 RELATION TO DROPOUT AND REGULARIZATION
The key point in successful pruning with NoiseOut is a higher correlation between neurons. This goal might seem to be in contradiction with techniques designed to avoid overfitting such as Dropout and Regularization. To investigate this, we pruned Lenet-5 in the presence and absence of these features and demonstrated the results in Figure 7. As it can be seen in this figure, Dropout helps the pruning process significantly, while L2-regularizations causes more variance. It seems that preventing co-adaptation of neurons using Dropout also intensifies the correlation between them, which helps NoiseOut to remove even more redundant neurons without accuracy loss.
5 CONCLUSION
In this paper, we have presented NoiseOut, a simple but effective pruning method to reduce the number of parameters in the dense layers of neural networks by removing neurons with correlated activation during training. We showed how adding noise outputs to the network could increase the correlation between neurons in the hidden layer and hence result to more efficient pruning. The experimental results on different networks and various datasets validate this approach, achieving significant compression rates without loss of accuracy.

This paper proposes and tests two ideas. (1) a method of pruning networks by identifying highly correlated neuron pairs, pruning one of the pair, and then modifying downstream weights to compensate for the removal (which works well if the removed neurons were highly correlated). (2) a method, dubbed NoiseOut, for increasing neuron correlation by adding auxiliary noise target outputs to the network during training.
The first idea (1) is fairly straightforward, and it is not clear if it has been tried before. It does seem to work.
The second idea (2) is of unclear value and seems to this reviewer that it may merely add a regularizing effect. Comments in this direction:
 - In Fig 4 (right), the constant and Gaussian treatments seem to produce the same effect in both networks, right? And the Binomial effect seems the same as No_Noise. If this is true, can we conclude that the NoiseOut targets are simply serving to regularize the network, that is, to reduce its capacity slightly?
 - To show whether this effect is true, one would need to compare to other methods of reducing the network capacity, for example: by reducing the number of neurons, by applying L2 regularization of various values, or by applying Dropout of various strengths. Fig 7 makes an attempt at this direction, but critically misses several comparison treatments: “Pruned without any regularization”, “Pruned with only L2”, and “Pruned with only DropOut”. Have these experiments been run? Can their results be included and used to produce plots like Fig 5 and Fig 7?
Without these comparisons, it seems impossible to conclude that NoiseOut does anything but provide similar regularization to DropOut or L2.
The combined ideas (1) + (2) DO produce a considerable reduction in parameters, but sadly the experiments and exposition are somewhat too lacking to really understand what is going on. With a little more work the paper could be quite interesting, but as is it should probably not be accepted.
Additional comments:
 - Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended.
 - Lowercase rho is used to indicate correlation but this is never actually specified, which is confusing for. Just state once that it indicates correlation.
 - How do these results compare to other pruning methods? No numerical comparison is attempted.


This paper presents an actor-critic deep reinforcement learning agent with experience replay that is stable, sample efficient, and performs remarkably well on challenging environments, including the discrete 57-game Atari domain and several continuous control problems. To achieve this, the paper introduces several innovations, including truncated importance sampling with bias correction, stochastic dueling network architectures, and a new trust region policy optimization method.
1 INTRODUCTION
Realistic simulated environments, where agents can be trained to learn a large repertoire of cognitive skills, are at the core of recent breakthroughs in AI (Bellemare et al., 2013; Mnih et al., 2015; Schulman et al., 2015a; Narasimhan et al., 2015; Mnih et al., 2016; Brockman et al., 2016; Oh et al., 2016). With richer realistic environments, the capabilities of our agents have increased and improved. Unfortunately, these advances have been accompanied by a substantial increase in the cost of simulation. In particular, every time an agent acts upon the environment, an expensive simulation step is conducted. Thus to reduce the cost of simulation, we need to reduce the number of simulation steps (i.e. samples of the environment). This need for sample efficiency is even more compelling when agents are deployed in the real world.
Experience replay (Lin, 1992) has gained popularity in deep Q-learning (Mnih et al., 2015; Schaul et al., 2016; Wang et al., 2016; Narasimhan et al., 2015), where it is often motivated as a technique for reducing sample correlation. Replay is actually a valuable tool for improving sample efficiency and, as we will see in our experiments, state-of-the-art deep Q-learning methods (Schaul et al., 2016; Wang et al., 2016) have been up to this point the most sample efficient techniques on Atari by a significant margin. However, we need to do better than deep Q-learning, because it has two important limitations. First, the deterministic nature of the optimal policy limits its use in adversarial domains. Second, finding the greedy action with respect to the Q function is costly for large action spaces.
Policy gradient methods have been at the heart of significant advances in AI and robotics (Silver et al., 2014; Lillicrap et al., 2015; Silver et al., 2016; Levine et al., 2015; Mnih et al., 2016; Schulman et al., 2015a; Heess et al., 2015). Many of these methods are restricted to continuous domains or to very specific tasks such as playing Go. The existing variants applicable to both continuous and discrete domains, such as the on-policy asynchronous advantage actor critic (A3C) of Mnih et al. (2016), are sample inefficient.
The design of stable, sample efficient actor critic methods that apply to both continuous and discrete action spaces has been a long-standing hurdle of reinforcement learning (RL). We believe this paper
is the first to address this challenge successfully at scale. More specifically, we introduce an actor critic with experience replay (ACER) that nearly matches the state-of-the-art performance of deep Q-networks with prioritized replay on Atari, and substantially outperforms A3C in terms of sample efficiency on both Atari and continuous control domains.
ACER capitalizes on recent advances in deep neural networks, variance reduction techniques, the off-policy Retrace algorithm (Munos et al., 2016) and parallel training of RL agents (Mnih et al., 2016). Yet, crucially, its success hinges on innovations advanced in this paper: truncated importance sampling with bias correction, stochastic dueling network architectures, and efficient trust region policy optimization.
On the theoretical front, the paper proves that the Retrace operator can be rewritten from our proposed truncated importance sampling with bias correction technique.
2 BACKGROUND AND PROBLEM SETUP
Consider an agent interacting with its environment over discrete time steps. At time step t, the agent observes the nx-dimensional state vector xt ∈ X ⊆ Rnx , chooses an action at according to a policy π(a|xt) and observes a reward signal rt ∈ R produced by the environment. We will consider discrete actions at ∈ {1, 2, . . . , Na} in Sections 3 and 4, and continuous actions at ∈ A ⊆ Rna in Section 5. The goal of the agent is to maximize the discounted return Rt = ∑ i≥0 γ
irt+i in expectation. The discount factor γ ∈ [0, 1) trades-off the importance of immediate and future rewards. For an agent following policy π, we use the standard definitions of the state-action and state only value functions:
Qπ(xt, at) = Ext+1:∞,at+1:∞ [Rt|xt, at] and V π(xt) = Eat [Qπ(xt, at)|xt] . Here, the expectations are with respect to the observed environment states xt and the actions generated by the policy π, where xt+1:∞ denotes a state trajectory starting at time t+ 1.
We also need to define the advantage function Aπ(xt, at) = Qπ(xt, at)− V π(xt), which provides a relative measure of value of each action since Eat [Aπ(xt, at)] = 0.
The parameters θ of the differentiable policy πθ(at|xt) can be updated using the discounted approximation to the policy gradient (Sutton et al., 2000), which borrowing notation from Schulman et al. (2015b), is defined as:
g = Ex0:∞,a0:∞
 ∑
t≥0
Aπ(xt, at)∇θ log πθ(at|xt)   . (1)
Following Proposition 1 of Schulman et al. (2015b), we can replaceAπ(xt, at) in the above expression with the state-action value Qπ(xt, at), the discounted return Rt, or the temporal difference residual rt + γV
π(xt+1) − V π(xt), without introducing bias. These choices will however have different variance. Moreover, in practice we will approximate these quantities with neural networks thus introducing additional approximation errors and biases. Typically, the policy gradient estimator using Rt will have higher variance and lower bias whereas the estimators using function approximation will have higher bias and lower variance. Combining Rt with the current value function approximation to minimize bias while maintaining bounded variance is one of the central design principles behind ACER.
To trade-off bias and variance, the asynchronous advantage actor critic (A3C) of Mnih et al. (2016) uses a single trajectory sample to obtain the following gradient approximation:
ĝa3c = ∑
t≥0
(( k−1∑
i=0
γirt+i ) + γkV πθv (xt+k)− V πθv (xt) ) ∇θ log πθ(at|xt). (2)
A3C combines both k-step returns and function approximation to trade-off variance and bias. We may think of V πθv (xt) as a policy gradient baseline used to reduce variance.
In the following section, we will introduce the discrete-action version of ACER. ACER may be understood as the off-policy counterpart of the A3C method of Mnih et al. (2016). As such, ACER builds on all the engineering innovations of A3C, including efficient parallel CPU computation.
ACER uses a single deep neural network to estimate the policy πθ(at|xt) and the value function V πθv (xt). (For clarity and generality, we are using two different symbols to denote the parameters of the policy and value function, θ and θv , but most of these parameters are shared in the single neural network.) Our neural networks, though building on the networks used in A3C, will introduce several modifications and new modules.
3 DISCRETE ACTOR CRITIC WITH EXPERIENCE REPLAY
Off-policy learning with experience replay may appear to be an obvious strategy for improving the sample efficiency of actor-critics. However, controlling the variance and stability of off-policy estimators is notoriously hard. Importance sampling is one of the most popular approaches for offpolicy learning (Meuleau et al., 2000; Jie & Abbeel, 2010; Levine & Koltun, 2013). In our context, it proceeds as follows. Suppose we retrieve a trajectory {x0, a0, r0, µ(·|x0), · · · , xk, ak, rk, µ(·|xk)}, where the actions have been sampled according to the behavior policy µ, from our memory of experiences. Then, the importance weighted policy gradient is given by:
ĝimp =
( k∏
t=0
ρt
) k∑
t=0
( k∑
i=0
γirt+i ) ∇θ log πθ(at|xt), (3)
where ρt = π(at|xt) µ(at|xt) denotes the importance weight. This estimator is unbiased, but it suffers from very high variance as it involves a product of many potentially unbounded importance weights. To prevent the product of importance weights from exploding, Wawrzyński (2009) truncates this product. Truncated importance sampling over entire trajectories, although bounded in variance, could suffer from significant bias.
Recently, Degris et al. (2012) attacked this problem by using marginal value functions over the limiting distribution of the process to yield the following approximation of the gradient:
gmarg = Ext∼β,at∼µ [ρt∇θ log πθ(at|xt)Qπ(xt, at)] , (4) where Ext∼β,at∼µ[·] is the expectation with respect to the limiting distribution β(x) = limt→∞ P (xt = x|x0, µ) with behavior policy µ. To keep the notation succinct, we will replace Ext∼β,at∼µ[·] with Extat [·] and ensure we remind readers of this when necessary. Two important facts about equation (4) must be highlighted. First, note that it depends on Qπ and not on Qµ, consequently we must be able to estimate Qπ. Second, we no longer have a product of importance weights, but instead only need to estimate the marginal importance weight ρt. Importance sampling in this lower dimensional space (over marginals as opposed to trajectories) is expected to exhibit lower variance.
Degris et al. (2012) estimateQπ in equation (4) using lambda returns: Rλt = rt+(1−λ)γV (xt+1)+ λγρt+1R λ t+1. This estimator requires that we know how to choose λ ahead of time to trade off bias and variance. Moreover, when using small values of λ to reduce variance, occasional large importance weights can still cause instability.
In the following subsection, we adopt the Retrace algorithm of Munos et al. (2016) to estimate Qπ. Subsequently, we propose an importance weight truncation technique to improve the stability of the off-policy actor critic of Degris et al. (2012), and introduce a computationally efficient trust region scheme for policy optimization. The formulation of ACER for continuous action spaces will require further innovations that are advanced in Section 5.
3.1 MULTI-STEP ESTIMATION OF THE STATE-ACTION VALUE FUNCTION
In this paper, we estimate Qπ(xt, at) using Retrace (Munos et al., 2016). (We also experimented with the related tree backup method of Precup et al. (2000) but found Retrace to perform better in practice.) Given a trajectory generated under the behavior policy µ, the Retrace estimator can be expressed recursively as follows1:
Qret(xt, at) = rt + γρ̄t+1[Q ret(xt+1, at+1)−Q(xt+1, at+1)] + γV (xt+1), (5)
1For ease of presentation, we consider only λ = 1 for Retrace.
where ρ̄t is the truncated importance weight, ρ̄t = min {c, ρt} with ρt = π(at|xt)µ(at|xt) , Q is the current value estimate of Qπ, and V (x) = Ea∼πQ(x, a). Retrace is an off-policy, return-based algorithm which has low variance and is proven to converge (in the tabular case) to the value function of the target policy for any behavior policy, see Munos et al. (2016).
The recursive Retrace equation depends on the estimate Q. To compute it, in discrete action spaces, we adopt a convolutional neural network with “two heads” that outputs the estimate Qθv (xt, at), as well as the policy πθ(at|xt). This neural representation is the same as in (Mnih et al., 2016), with the exception that we output the vector Qθv (xt, at) instead of the scalar Vθv (xt). The estimate Vθv (xt) can be easily derived by taking the expectation of Qθv under πθ.
To approximate the policy gradient gmarg, ACER uses Qret to estimate Qπ. As Retrace uses multistep returns, it can significantly reduce bias in the estimation of the policy gradient 2.
To learn the critic Qθv (xt, at), we again use Q ret(xt, at) as a target in a mean squared error loss and update its parameters θv with the following standard gradient:
(Qret(xt, at)−Qθv (xt, at))∇θvQθv (xt, at)). (6) Because Retrace is return-based, it also enables faster learning of the critic. Thus the purpose of the multi-step estimator Qret in our setting is twofold: to reduce bias in the policy gradient, and to enable faster learning of the critic, hence further reducing bias.
3.2 IMPORTANCE WEIGHT TRUNCATION WITH BIAS CORRECTION
The marginal importance weights in Equation (4) can become large, thus causing instability. To safe-guard against high variance, we propose to truncate the importance weights and introduce a correction term via the following decomposition of gmarg:
gmarg =Extat [ρt∇θlog πθ(at|xt)Qπ(xt, at)]
=Ext [ Eat[ρ̄t∇θlog πθ(at|xt)Qπ(xt, at)]+E
a∼π ([ ρt(a)− c ρt(a) ]
+
∇θlog πθ(a|xt)Qπ(xt, a) )] ,(7)
where ρ̄t = min {c, ρt} with ρt = π(at|xt)µ(at|xt) as before. We have also introduced the notation ρt(a) =
π(a|xt) µ(a|xt) , and [x]+ = x if x > 0 and it is zero otherwise. We remind readers that the above
expectations are with respect to the limiting state distribution under the behavior policy: xt ∼ β and at ∼ µ. The clipping of the importance weight in the first term of equation (7) ensures that the variance of the gradient estimate is bounded. The correction term (second term in equation (7)) ensures that our estimate is unbiased. Note that the correction term is only active for actions such that ρt(a) > c. In particular, if we choose a large value for c, the correction term only comes into effect when the variance of the original off-policy estimator of equation (4) is very high. When this happens, our decomposition has the nice property that the truncated weight in the first term is at most c while the correction weight [ ρt(a)−c ρt(a) ] + in the second term is at most 1.
We model Qπ(xt, a) in the correction term with our neural network approximation Qθv (xt, at). This modification results in what we call the truncation with bias correction trick, in this case applied to the function ∇θ log πθ(at|xt)Qπ(xt, at):
ĝmarg =Ext [ Eat [ ρ̄t∇θlog πθ(at|xt)Qret(xt, at) ] +E a∼π ([ ρt(a)− c ρt(a) ]
+
∇θlog πθ(a|xt)Qθv (xt, a) )] .(8)
Equation (8) involves an expectation over the stationary distribution of the Markov process. We can however approximate it by sampling trajectories {x0, a0, r0, µ(·|x0), · · · , xk, ak, rk, µ(·|xk)}
2An alternative to Retrace here is Q(λ) with off-policy corrections (Harutyunyan et al., 2016) which we discuss in more detail in Appendix B.
generated from the behavior policy µ. Here the terms µ(·|xt) are the policy vectors. Given these trajectories, we can compute the off-policy ACER gradient:
ĝacert = ρ̄t∇θ log πθ(at|xt)[Qret(xt, at)− Vθv (xt)]
+ E a∼π ([ ρt(a)− c ρt(a) ]
+
∇θ log πθ(a|xt)[Qθv (xt, a)− Vθv (xt)] ) . (9)
In the above expression, we have subtracted the classical baseline Vθv (xt) to reduce variance.
It is interesting to note that, when c = ∞, (9) recovers (off-policy) policy gradient up to the use of Retrace. When c = 0, (9) recovers an actor critic update that depends entirely on Q estimates. In the continuous control domain, (9) also generalizes Stochastic Value Gradients if c = 0 and the reparametrization trick is used to estimate its second term (Heess et al., 2015).
3.3 EFFICIENT TRUST REGION POLICY OPTIMIZATION
The policy updates of actor-critic methods do often exhibit high variance. Hence, to ensure stability, we must limit the per-step changes to the policy. Simply using smaller learning rates is insufficient as they cannot guard against the occasional large updates while maintaining a desired learning speed. Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a) provides a more adequate solution.
Schulman et al. (2015a) approximately limit the difference between the updated policy and the current policy to ensure safety. Despite the effectiveness of their TRPO method, it requires repeated computation of Fisher-vector products for each update. This can prove to be prohibitively expensive in large domains.
In this section we introduce a new trust region policy optimization method that scales well to large problems. Instead of constraining the updated policy to be close to the current policy (as in TRPO), we propose to maintain an average policy network that represents a running average of past policies and forces the updated policy to not deviate far from this average.
We decompose our policy network in two parts: a distribution f , and a deep neural network that generates the statistics φθ(x) of this distribution. That is, given f , the policy is completely characterized by the network φθ: π(·|x) = f(·|φθ(x)). For example, in the discrete domain, we choose f to be the categorical distribution with a probability vector φθ(x) as its statistics. The probability vector is of course parameterised by θ.
We denote the average policy network as φθa and update its parameters θa “softly” after each update to the policy parameter θ: θa ← αθa + (1− α)θ. Consider, for example, the ACER policy gradient as defined in Equation (9), but with respect to φ:
ĝacert = ρ̄t∇φθ(xt) log f(at|φθ(x))[Qret(xt, at)− Vθv (xt)]
+ E a∼π ([ ρt(a)− c ρt(a) ]
+
∇φθ(xt) log f(at|φθ(x))[Qθv (xt, a)− Vθv (xt)] ) . (10)
Given the averaged policy network, our proposed trust region update involves two stages. In the first stage, we solve the following optimization problem with a linearized KL divergence constraint:
minimize z
1 2 ‖ĝacert − z‖22
subject to ∇φθ(xt)DKL [f(·|φθa(xt))‖f(·|φθ(xt))] T z ≤ δ
(11)
Since the constraint is linear, the overall optimization problem reduces to a simple quadratic programming problem, the solution of which can be easily derived in closed form using the KKT conditions. Letting k = ∇φθ(xt)DKL [f(·|φθa(xt)‖f(·|φθ(xt)], the solution is:
z∗ = ĝacert −max {
0, kT ĝacert − δ ‖k‖22
} k (12)
This transformation of the gradient has a very natural form. If the constraint is satisfied, there is no change to the gradient with respect to φθ(xt). Otherwise, the update is scaled down in the direction
of k, thus effectively lowering rate of change between the activations of the current policy and the average policy network.
In the second stage, we take advantage of back-propagation. Specifically, the updated gradient with respect to φθ, that is z∗, is back-propagated through the network to compute the derivatives with respect to the parameters. The parameter updates for the policy network follow from the chain rule: ∂φθ(x) ∂θ z ∗.
The trust region step is carried out in the space of the statistics of the distribution f , and not in the space of the policy parameters. This is done deliberately so as to avoid an additional back-propagation step through the policy network.
We would like to remark that the algorithm advanced in this section can be thought of as a general strategy for modifying the backward messages in back-propagation so as to stabilize the activations.
Instead of a trust region update, one could alternatively add an appropriately scaled KL cost to the objective function as proposed by Heess et al. (2015). This approach, however, is less robust to the choice of hyper-parameters in our experience.
The ACER algorithm results from a combination of the above ideas, with the precise pseudo-code appearing in Appendix A. A master algorithm (Algorithm 1) calls ACER on-policy to perform updates and propose trajectories. It then calls ACER off-policy component to conduct several replay steps. When on-policy, ACER effectively becomes a modified version of A3C where Q instead of V baselines are employed and trust region optimization is used.
4 RESULTS ON ATARI
We use the Arcade Learning Environment of Bellemare et al. (2013) to conduct an extensive evaluation. We deploy one single algorithm and network architecture, with fixed hyper-parameters, to learn to play 57 Atari games given only raw pixel observations and game rewards. This task is highly demanding because of the diversity of games, and high-dimensional pixel-level observations.
Our experimental setup uses 16 actor-learner threads running on a single machine with no GPUs. We adopt the same input pre-processing and network architecture as Mnih et al. (2015). Specifically, the network consists of a convolutional layer with 32 8× 8 filters with stride 4 followed by another convolutional layer with 64 4× 4 filters with stride 2, followed by a final convolutional layer with 64 3× 3 filters with stride 1, followed by a fully-connected layer of size 512. Each of the hidden layers is followed by a rectifier nonlinearity. The network outputs a softmax policy and Q values.
When using replay, we add to each thread a replay memory that is up to 50 000 frames in size. The total amount of memory used across all threads is thus similar in size to that of DQN (Mnih et al., 2015). For all Atari experiments, we use a single learning rate adopted from an earlier implementation of A3C without further tuning. We do not anneal the learning rates over the course of training as in Mnih et al. (2016). We otherwise adopt the same optimization procedure as in Mnih et al. (2016). Specifically, we adopt entropy regularization with weight 0.001, discount the rewards with γ = 0.99, and perform updates every 20 steps (k = 20 in the notation of Section 2). In all our experiments with experience replay, we use importance weight truncation with c = 10. We consider training ACER both with and without trust region updating as described in Section 3.3. When trust region updating is used, we use δ = 1 and α = 0.99 for all experiments.
To compare different agents, we adopt as our metric the median of the human normalized score over all 57 games. The normalization is calculated such that, for each game, human scores and random scores are evaluated to 1, and 0 respectively. The normalized score for a given game at time t is computed as the average normalized score over the past 1 million consecutive frames encountered until time t. For each agent, we plot its cumulative maximum median score over time. The result is summarized in Figure 1.
The four colors in Figure 1 correspond to four replay ratios (0, 1, 4 and 8) with a ratio of 4 meaning that we use the off-policy component of ACER 4 times after using the on-policy component (A3C). That is, a replay ratio of 0 means that we are using A3C. The solid and dashed lines represent ACER with and without trust region updating respectively. The gray and black curves are the original DQN (Mnih et al., 2015) and Prioritized Replay agent of Schaul et al. (2016) agents respectively.
As shown on the left panel of Figure 1, replay significantly increases data efficiency. We observe that when using the trust region optimizer, the average reward as a function of the number of environmental steps increases with the ratio of replay. This increase has diminishing returns, but with enough replay, ACER can match the performance of the best DQN agents. Moreover, it is clear that the off-policy actor critics (ACER) are much more sample efficient than their on-policy counterpart (A3C).
The right panel of Figure 1 shows that ACER agents perform similarly to A3C when measured by wall clock time. Thus, in this case, it is possible to achieve better data-efficiency without necessarily compromising on computation time. In particular, ACER with a replay ratio of 4 is an appealing alternative to either the prioritized DQN agent or A3C.
5 CONTINUOUS ACTOR CRITIC WITH EXPERIENCE REPLAY
Retrace requires estimates of both Q and V , but we cannot easily integrate over Q to derive V in continuous action spaces. In this section, we propose a solution to this problem in the form of a novel representation for RL, as well as modifications necessary for trust region updating.
5.1 POLICY EVALUATION
Retrace provides a target for learning Qθv , but not for learning Vθv . We could use importance sampling to compute Vθv given Qθv , but this estimator has high variance.
We propose a new architecture which we call Stochastic Dueling Networks (SDNs), inspired by the Dueling networks of Wang et al. (2016), which is designed to estimate both V π and Qπ off-policy while maintaining consistency between the two estimates. At each time step, an SDN outputs a stochastic estimate Q̃θv of Q π and a deterministic estimate Vθv of V π , such that
Q̃θv (xt, at) ∼ Vθv (xt) +Aθv (xt, at)− 1
n
n∑
i=1
Aθv (xt, ui), and ui ∼ πθ(·|xt) (13)
where n is a parameter, see Figure 2. The two estimates are consistent in the sense that Ea∼π(·|xt) [ Eu1:n∼π(·|xt) ( Q̃θv (xt, a) )] = Vθv (xt). Furthermore, we can learn about V π by learn-
ing Q̃θv . To see this, assume we have learned Q π perfectly such that Eu1:n∼π(·|xt) ( Q̃θv (xt, at) ) = Qπ(xt, at), then Vθv (xt) = Ea∼π(·|xt) [ Eu1:n∼π(·|xt) ( Q̃θv (xt, a) )] = Ea∼π(·|xt) [Qπ(xt, a)] =
V π(xt). Therefore, a target on Q̃θv (xt, at) also provides an error signal for updating Vθv .
In addition to SDNs, however, we also construct the following novel target for estimating V π:
V target(xt) = min { 1, π(at|xt) µ(at|xt) }( Qret(xt, at)−Qθv (xt, at) ) + Vθv (xt). (14)
The above target is also derived via the truncation and bias correction trick; for more details, see Appendix D.
Finally, when estimating Qret in continuous domains, we implement a slightly different formulation of the truncated importance weights ρ̄t = min { 1, ( π(at|xt) µ(at|xt) ) 1 d } , where d is the dimensionality of
the action space. Although not essential, we have found this formulation to lead to faster learning.
5.2 TRUST REGION UPDATING
To adopt the trust region updating scheme (Section 3.3) in the continuous control domain, one simply has to choose a distribution f and a gradient specification ĝacert suitable for continuous action spaces.
For the distribution f , we choose Gaussian distributions with fixed diagonal covariance and mean φθ(x).
To derive ĝacert in continuous action spaces, consider the ACER policy gradient for the stochastic dueling network, but with respect to φ:
gacert = Ext [ Eat [ ρ̄t∇φθ(xt) log f(at|φθ(xt))(Qopc(xt, at)− Vθv (xt)) ]
+ E a∼π ([ ρt(a)− c ρt(a) ]
+
(Q̃θv (xt, a)− Vθv (xt))∇φθ(xt) log f(a|φθ(xt)) )] . (15)
In the above definition, we are using Qopc instead of Qret. Here, Qopc(xt, at) is the same as Retrace with the exception that the truncated importance ratio is replaced with 1 (Harutyunyan et al., 2016). Please refer to Appendix B an expanded discussion on this design choice. Given an observation xt, we can sample a′t ∼ πθ(·|xt) to obtain the following Monte Carlo approximation
ĝacert = ρ̄t∇φθ(xt) log f(at|φθ(xt))(Qopc(xt, at)− Vθv (xt))
+
[ ρt(a ′ t)− c
ρt(a′t)
]
+
(Q̃θv (xt, a ′ t)− Vθv (xt))∇φθ(xt) log f(a′t|φθ(xt)). (16)
Given f and ĝacert , we apply the same steps as detailed in Section 3.3 to complete the update.
The precise pseudo-code of ACER algorithm for continuous spaces results is presented in Appendix A.
6 RESULTS ON MUJOCO
We evaluate our algorithms on 6 continuous control tasks, all of which are simulated using the MuJoCo physics engine (Todorov et al., 2012). For descriptions of the tasks, please refer to Appendix E.1. Briefly, the tasks with action dimensionality in brackets are: cartpole (1D), reacher (3D), cheetah (6D), fish (5D), walker (6D) and humanoid (21D). These tasks are illustrated in Figure 3.
To benchmark ACER for continuous control, we compare it to its on-policy counterpart both with and without trust region updating. We refer to these two baselines as A3C and Trust-A3C. Additionally, we also compare to a baseline with replay where we truncate the importance weights over trajectories as in (Wawrzyński, 2009). For a detailed description of this baseline, please refer to Appendix E. Again, we run this baseline both with and without trust region updating, and refer to these choices as Trust-TIS and TIS respectively. Last but not least, we refer to our proposed approach with SDN and trust region updating as simply ACER. All five setups are implemented in the asynchronous A3C framework.
All the aforementioned setups share the same network architecture that computes the policy and state values. We maintain an additional small network that computes the stochastic A values in the case of ACER. We use n = 5 (using the notation in Equation (13)) in all SDNs. Instead of mixing on-policy and replay learning as done in the Atari domain, ACER for continuous actions is entirely off-policy, with experiences generated from the simulator (4 times on average). When using replay, we add to each thread a replay memory that is 5, 000 frames in size and perform updates every 50 steps (k = 50 in the notation of Section 2). The rate of the soft updating (α as in Section 3.3) is set to 0.995 in all setups involving trust region updating. The truncation threshold c is set to 5 for ACER.
We use diagonal Gaussian policies with fixed diagonal covariances where the diagonal standard deviation is set to 0.3. For all setups, we sample the learning rates log-uniformly in the range [10−4, 10−3.3]. For setups involving trust region updating, we also sample δ uniformly in the range [0.1, 2]. With all setups, we use 30 sampled hyper-parameter settings.
The empirical results for all continuous control tasks are shown Figure 3, where we show the mean and standard deviation of the best 5 out of 30 hyper-parameter settings over which we searched 3. For sensitivity analyses with respect to the hyper-parameters, please refer to Figures 5 and 6 in the Appendix.
In continuous control, ACER outperforms the A3C and truncated importance sampling baselines by a very significant margin.
Here, we also find that the proposed trust region optimization method can result in huge improvements over the baselines. The high-dimensional continuous action policies are much harder to optimize than the small discrete action policies in Atari, and hence we observe much higher gains for trust region optimization in the continuous control domains. In spite of the improvements brought in by trust region optimization, ACER still outperforms all other methods, specially in higher dimensions.
6.1 ABLATIONS
To further tease apart the contributions of the different components of ACER, we conduct an ablation analysis where we individually remove Retrace / Q(λ) off-policy correction, SDNs, trust region, and truncation with bias correction from the algorithm. As shown in Figure 4, Retrace and offpolicy correction, SDNs, and trust region are critical: removing any one of them leads to a clear deterioration of the performance. Truncation with bias correction did not alter the results in the Fish and Walker2d tasks. However, in Humanoid, where the dimensionality of the action space is much higher, including truncation and bias correction brings a significant boost which makes the originally kneeling humanoid stand. Presumably, the high dimensionality of the action space increases the variance of the importance weights which makes truncation with bias correction important. For more details on the experimental setup please see Appendix E.4.
7 THEORETICAL ANALYSIS
Retrace is a very recent development in reinforcement learning. In fact, this work is the first to consider Retrace in the policy gradients setting. For this reason, and given the core role that Retrace plays in ACER, it is valuable to shed more light on this technique. In this section, we will prove that Retrace can be interpreted as an application of the importance weight truncation and bias correction trick advanced in this paper.
Consider the following equation:
Qπ(xt, at) = Ext+1at+1 [rt + γρt+1Qπ(xt+1, at+1)] . (17)
If we apply the weight truncation and bias correction trick to the above equation we obtain
Qπ(xt, at) = Ext+1at+1 [ rt + γρ̄t+1Q
π(xt+1, at+1) + γ E a∼π ([ ρt+1(a)− c ρt+1(a) ]
+
Qπ(xt+1, a) )] .
(18) By recursively expanding Qπ as in Equation (18), we can represent Qπ(x, a) as:
Qπ(x, a) = Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Qπ(xt+1, b) ))  . (19)
The expectation Eµ is taken over trajectories starting from x with actions generated with respect to µ. When Qπ is not available, we can replace it with our current estimate Q to get a return-based
3 For videos of the policies learned with ACER, please see: https://www.youtube.com/watch?v= NmbeQYoVv5g&list=PLkmHIkhlFjiTlvwxEnsJMs3v7seR5HSP-.
esitmate of Qπ . This operation also defines an operator:
BQ(x, a) = Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Q(xt+1, b)
))  . (20)
In the following proposition, we show that B is a contraction operator with a unique fixed point Qπ and that it is equivalent to the Retrace operator. Proposition 1. The operator B is a contraction operator such that ‖BQ−Qπ‖∞ ≤ γ‖Q−Qπ‖∞ and B is equivalent to Retrace.
The above proposition not only shows an alternative way of arriving at the same operator, but also provides a different proof of contraction for Retrace. Please refer to Appendix C for the regularization conditions and proof of the above proposition.
Finally, B, and therefore Retrace, generalizes both the Bellman operator T π and importance sampling. Specifically, when c = 0, B = T π and when c = ∞, B recovers importance sampling; see Appendix C.
8 CONCLUDING REMARKS
We have introduced a stable off-policy actor critic that scales to both continuous and discrete action spaces. This approach integrates several recent advances in RL in a principle manner. In addition, it integrates three innovations advanced in this paper: truncated importance sampling with bias correction, stochastic dueling networks and an efficient trust region policy optimization method.
We showed that the method not only matches the performance of the best known methods on Atari, but that it also outperforms popular techniques on several continuous control problems.
The efficient trust region optimization method advanced in this paper performs remarkably well in continuous domains. It could prove very useful in other deep learning domains, where it is hard to stabilize the training process.
ACKNOWLEDGMENTS
We are very thankful to Marc Bellemare, Jascha Sohl-Dickstein, and Sébastien Racaniere for proofreading and valuable suggestions.
A ACER PSEUDO-CODE FOR DISCRETE ACTIONS
Algorithm 1 ACER for discrete actions (master algorithm) // Assume global shared parameter vectors θ and θv . // Assume ratio of replay r. repeat
Call ACER on-policy, Algorithm 2. n← Possion(r) for i ∈ {1, · · · , n} do
Call ACER off-policy, Algorithm 2. end for
until Max iteration or time reached.
Algorithm 2 ACER for discrete actions Reset gradients dθ ← 0 and dθv ← 0. Initialize parameters θ′ ← θ and θ′v ← θv . if not On-Policy then
Sample the trajectory {x0, a0, r0, µ(·|x0), · · · , xk, ak, rk, µ(·|xk)} from the replay memory. else
Get state x0 end if for i ∈ {0, · · · , k} do
Compute f(·|φθ′(xi)), Qθ′v (xi, ·) and f(·|φθa(xi)). if On-Policy then
Perform ai according to f(·|φθ′(xi)) Receive reward ri and new state xi+1 µ(·|xi)← f(·|φθ′(xi))
end if ρ̄i ← min { 1,
f(ai|φθ′ (xi)) µ(ai|xi)
} .
end for
Qret ← { 0 for terminal xk∑ aQθ′v (xk, a)f(a|φθ′(xk)) otherwise for i ∈ {k − 1, · · · , 0} do Qret ← ri + γQret Vi ← ∑ aQθ′v (xi, a)f(a|φθ′(xi))
Computing quantities needed for trust region updating:
g ← min {c, ρi(ai)}∇φθ′ (xi) log f(ai|φθ′(xi))(Q ret − Vi)
+ ∑ a [ 1− c ρi(a) ] + f(a|φθ′(xi))∇φθ′ (xi) log f(a|φθ′(xi))(Qθ′v (xi, ai)− Vi)
k ← ∇φθ′ (xi)DKL [f(·|φθa(xi)‖f(·|φθ′(xi)]
Accumulate gradients wrt θ′: dθ′ ← dθ′ + ∂φθ′ (xi) ∂θ′
( g −max { 0, k
T g−δ ‖k‖22
} k )
Accumulate gradients wrt θ′v: dθv ← dθv +∇θ′v (Q ret −Qθ′v (xi, a)) 2 Update Retrace target: Qret ← ρ̄i ( Qret −Qθ′v (xi, ai) ) + Vi
end for Perform asynchronous update of θ using dθ and of θv using dθv . Updating the average policy network: θa ← αθa + (1− α)θ
B Q(λ) WITH OFF-POLICY CORRECTIONS
Given a trajectory generated under the behavior policy µ, the Q(λ) with off-policy corrections estimator (Harutyunyan et al., 2016) can be expressed recursively as follows:
Qopc(xt, at) = rt + γ[Q opc(xt+1, at+1)−Q(xt+1, at+1)] + γV (xt+1). (21)
Notice that Qopc(xt, at) is the same as Retrace with the exception that the truncated importance ratio is replaced with 1.
Algorithm 3 ACER for Continuous Actions Reset gradients dθ ← 0 and dθv ← 0. Initialize parameters θ′ ← θ and θ′v ← θv . Sample the trajectory {x0, a0, r0, µ(·|x0), · · · , xk, ak, rk, µ(·|xk)} from the replay memory. for i ∈ {0, · · · , k} do
Compute f(·|φθ′(xi)), Vθ′v (xi), Q̃θ′v (xi, ai), and f(·|φθa(xi)). Sample a′i ∼ f(·|φθ′(xi)) ρi ← f(ai|φθ′ (xi))µ(ai|xi) and ρ ′ i ← f(a′i|φθ′ (xi)) µ(a′i|xi)
ci ← min { 1, (ρi) 1 d } .
end for
Qret ← { 0 for terminal xk Vθ′v (xk) otherwise Qopc ← Qret for i ∈ {k − 1, · · · , 0} do Qret ← ri + γQret Qopc ← ri + γQopc Computing quantities needed for trust region updating:
g ← min {c, ρi}∇φθ′ (xi) log f(ai|φθ′(xi)) ( Qopc(xi, ai)− Vθ′v (xi) ) + [ 1− c
ρ′i ] + (Q̃θ′v (xi, a ′ i)− Vθ′v (xi))∇φθ′ (xi) log f(a ′ i|φθ′(xi))
k ← ∇φθ′ (xi)DKL [f(·|φθa(xi)‖f(·|φθ′(xi)]
Accumulate gradients wrt θ: dθ ← dθ + ∂φθ′ (xi) ∂θ′
( g −max { 0, k
T g−δ ‖k‖22
} k )
Accumulate gradients wrt θ′v: dθv ← dθv + (Qret − Q̃θ′v (xi, ai))∇θ′v Q̃θ′v (xi, ai) dθv ← dθv + min {1, ρi} ( Qret(xt, ai)− Q̃θ′v (xt, ai) ) ∇θ′vVθ′v (xi)
Update Retrace target: Qret ← ci ( Qret − Q̃θ′v (xi, ai) ) + Vθ′v (xi)
Update Retrace target: Qopc ← ( Qopc − Q̃θ′v (xi, ai) ) + Vθ′v (xi)
end for Perform asynchronous update of θ using dθ and of θv using dθv . Updating the average policy network: θa ← αθa + (1− α)θ
Because of the lack of the truncated importance ratio, the operator defined by Qopc is only a contraction if the target and behavior policies are close to each other (Harutyunyan et al., 2016). Q(λ) with off-policy corrections is therefore less stable compared to Retrace and unsafe for policy evaluation.
Qopc, however, could better utilize the returns as the traces are not cut by the truncated importance weights. As a result,Qopc could be used efficiently to estimateQπ in policy gradient (e.g. in Equation (16)). In our continuous control experiments, we have found that Qopc leads to faster learning.
C RETRACE AS TRUNCATED IMPORTANCE SAMPLING WITH BIAS CORRECTION
For the purpose of proving proposition 1, we assume our environment to be a Markov Decision Process (X ,A, γ, P, r). We restrict X to be a finite state space. For notational simplicity, we also restrict A to be a finite action space. P : X ,A → X defines the state transition probabilities and r : X ,A → [−RMAX, RMAX] defines a reward function. Finally, γ ∈ [0, 1) is the discount factor.
Proof of proposition 1. First we show that B is a contraction operator. |BQ(x, a)−Qπ(x, a)|
= ∣∣∣∣∣∣ Eµ  ∑
t≥0
γt
( t∏
i=1
ρ̄i )( γ E b∼π ([ ρt+1(b)− c ρt+1(b) ]
+ (Q(xt+1, b)−Qπ(xt+1, b)) ))  ∣∣∣∣∣∣
≤ Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )[ γ E b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
|Q(xt+1, b)−Qπ(xt+1, b)| )]
≤ Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( γ(1− P̄t+1) sup
b |Q(xt+1, b)−Qπ(xt+1, b)|
)  (22)
Where P̄t+1 = 1− E b∼π [[ ρt+1(b)−c ρt+1(b) ] + ] = E b∼µ [ρ̄t+1(b)]. The last inequality in the above equation is due to Hölder’s inequality.
(22) ≤ sup x,b |Q(x, b)−Qπ(x, b)|Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i ) ( γ(1− P̄t+1) )
= sup x,b |Q(x, b)−Qπ(x, b)|Eµ
 γ ∑
t≥0
γt
( t∏
i=1
ρ̄i
) − ∑
t≥0
γt
( t∏
i=1
ρ̄i ) ( γP̄t+1 )
= sup x,b |Q(x, b)−Qπ(x, b)|Eµ
 γ ∑
t≥0
γt
( t∏
i=1
ρ̄i
) − ∑
t≥0
γt+1
( t+1∏
i=1
ρ̄i
)
= sup x,b |Q(x, b)−Qπ(x, b)| (γC − (C − 1))
whereC = ∑ t≥0 γ
t (∏t
i=1 ρ̄i ) . SinceC ≥∑0t=0 γt (∏t i=1 ρ̄i ) = 1, we have that γC−(C−1) ≤
γ. Therefore, we have shown that B is a contraction operator. Now we show that B is the same as Retrace. By apply the trunction and bias correction trick, we have
E b∼π [Q(xt+1, b)] = E b∼µ [ρ̄t+1(b)Q(xt+1, b)] + E b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Q(xt+1, b)
) . (23)
By adding and subtracting the two sides of Equation (23) inside the summand of Equation (20), we have BQ(x, a) = Eµ [∑
t≥0
γt ( t∏
i=1
ρ̄i )[ rt + γ E
b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Q(xt+1, b)
) +γ E
b∼π [Q(xt+1, b)]
−γ E b∼µ [ρ̄t+1(b)Q(xt+1, b)]− γ E b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Q(xt+1, b)
)]]
= Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π [Q(xt+1, b)]− γ E b∼µ [ρ̄t+1(b)Q(xt+1, b)]
)
= Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π [Q(xt+1, b)]− γρ̄t+1Q(xt+1, at+1)
)
= Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π [Q(xt+1, b)]−Q(xt, at)
) +Q(x, a) = RQ(x, a)
In the remainder of this appendix, we show that B generalizes both the Bellman operator and importance sampling. First, we reproduce the definition of B:
BQ(x, a) = Eµ
 ∑
t≥0
γt
( t∏
i=1
ρ̄i )( rt + γ E
b∼π ([ ρt+1(b)− c ρt+1(b) ]
+
Q(xt+1, b)
))  .
When c = 0, we have that ρ̄i = 0 ∀i. Therefore only the first summand of the sum remains:
BQ(x, a) = Eµ [ rt + γ E
b∼π (Q(xt+1, b))
] .
In this case B = T . When c =∞, the compensation term disappears and ρ̄i = ρi ∀i:
BQ(x, a) = Eµ
 ∑
t≥0
γt
( t∏
i=1
ρi )( rt + γ E
b∼π (0×Q(xt+1, b))
)  = Eµ  ∑
t≥0
γt
( t∏
i=1
ρi ) rt   .
In this case B is the same operator defined by importance sampling.
D DERIVATION OF V target
By using the truncation and bias correction trick, we can derive the following:
V π(xt) = E a∼µ
[ min { 1, π(a|xt) µ(a|xt) } Qπ(xt, a) ] + E a∼π ([ ρt(a)− 1 ρt(a) ]
+
Qπ(xt+1, a) ) .
We, however, cannot use the above equation as a target as we do not have access to Qπ . To derive a target, we can take a Monte Carlo approximation of the first expectation in the RHS of the above equation and replace the first occurrence of Qπ with Qret and the second with our current neural net approximation Qθv (xt, ·):
V targetpre (xt) := min { 1, π(at|xt) µ(at|xt) } Qret(xt, at) + E a∼π ([ ρt(a)− 1 ρt(a) ]
+
Qθv (xt, a)
) . (24)
Through the truncation and bias correction trick again, we have the following identity:
E a∼π [Qθv (xt, a)] = E a∼µ
[ min { 1, π(a|xt) µ(a|xt) } Qθv (xt, a) ] + E a∼π ([ ρt(a)− 1 ρt(a) ]
+
Qθv (xt, a)
) . (25)
Adding and subtracting both sides of Equation (25) to the RHS of (24) while taking a Monte Carlo approximation, we arrive at V target(xt):
V target(xt) := min { 1, π(at|xt) µ(at|xt) }( Qret(xt, at)−Qθv (xt, at) ) + Vθv (xt).
E CONTINUOUS CONTROL EXPERIMENTS
E.1 DESCRIPTION OF THE CONTINUOUS CONTROL PROBLEMS
Our continuous control tasks were simulated using the MuJoCo physics engine (Todorov et al. (2012)). For all experiments we considered an episodic setup with an episode length of T = 500 steps and a discount factor of 0.99.
Cartpole swingup This is an instance of the classic cart-pole swing-up task. It consists of a pole attached to a cart running on a finite track. The agent is required to balance the pole near the center of the track by applying a force to the cart only. An episode starts with the pole at a random angle and zero velocity. A reward zero is given except when the pole is approximately upright (within ±5 deg) and the track approximately in the center of the track (±0.05) for a track length of 2.4. The observations include position and velocity of the cart, angle and angular velocity of the pole. a sine/cosine of the angle, the position of the tip of the pole, and Cartesian velocities of the pole. The dimension of the action space is 1.
Reacher3 The agent needs to control a planar 3-link robotic arm in order to minimize the distance between the end effector of the arm and a target. Both arm and target position are chosen randomly at the beginning of each episode. The reward is zero except when the tip of the arm is within 0.05 of the target, where it is one. The 8-dimensional observation consists of the angles and angular velocity of all joints as well as the displacement between target and the end effector of the arm. The 3-dimensional action are the torques applied to the joints.
Cheetah The Half-Cheetah (Wawrzyński (2009); Heess et al. (2015)) is a planar locomotion task where the agent is required to control a 9-DoF cheetah-like body (in the vertical plane) to move in the direction of the x-axis as quickly as possible. The reward is given by the velocity along the x-axis and a control cost: r = vx + 0.1‖a‖2. The observation vector consists of the z-position of the torso and its x, z velocities as well as the joint angles and angular velocities. The action dimension is 6.
Fish The goal of this task is to control a 13-DoF fish-like body to swim to a random target in 3D space. The reward is given by the distance between the head of the fish and the target, a small penalty for the body not being upright, and a control cost. At the beginning of an episode the fish is initialized facing in a random direction relative to the target. The 24-dimensional observation is given by the displacement between the fish and the target projected onto the torso coordinate frame, the joint angles and velocities, the cosine of the angle between the z-axis of the torso and the world z-axis, and the velocities of the torso in the torso coordinate frame. The 5-dimensional actions control the position of the side fins and the tail.
Walker The 9-DoF planar walker is inspired by (Schulman et al. (2015a)) and is required to move forward along the x-axis as quickly as possible without falling. The reward consists of the x-velocity of the torso, a quadratic control cost, and terms that penalize deviations of the torso from the preferred height and orientation (i.e. terms that encourage the walker to stay standing and upright). The 24-dimensional observation includes the torso height, velocities of all DoFs, as well as sines and cosines of all body orientations in the x-z plane. The 6-dimensional action controls the torques applied at the joints. Episodes are terminated early with a negative reward when the torso exceeds upper and lower limits on its height and orientation.
Humanoid The humanoid is a 27 degrees-of-freedom body with 21 actuators (21 action dimensions). It is initialized lying on the ground in a random configuration and the task requires it to achieve a standing position. The reward function penalizes deviations from the height of the head when standing, and includes additional terms that encourage upright standing, as well as a quadratic action penalty. The 94 dimensional observation contains information about joint angles and velocities and several derived features reflecting the body’s pose.
E.2 UPDATE EQUATIONS OF THE BASELINE TIS
The baseline TIS follows the following update equations,
updates to the policy: min { 5, ( k−1∏
i=0
ρt+i
)}[ k−1∑
i=0
γirt+i + γ kVθv (xk+t)− Vθv (xt) ] ∇θ log πθ(at|xt),
updates to the value: min { 5, ( k−1∏
i=0
ρt+i
)}[ k−1∑
i=0
γirt+i + γ kVθv (xk+t)− Vθv (xt) ] ∇θvVθv (xt).
The baseline Trust-TIS is appropriately modified according to the trust region update described in Section 3.3.
E.3 SENSITIVITY ANALYSIS
In this section, we assess the sensitivity of ACER to hyper-parameters. In Figures 5 and 6, we show, for each game, the final performance of our ACER agent versus the choice of learning rates, and the trust region constraint δ respectively.
Note, as we are doing random hyper-parameter search, each learning rate is associated with a random δ and vice versa. It is therefore difficult to tease out the effect of either hyper-parameter independently.
We observe, however, that ACER is not very sensitive to the hyper-parameters overall. In addition, smaller δ’s do not seem to adversely affect the final performance while larger δ’s do in domains of higher action dimensionality. Similarly, smaller learning rates perform well while bigger learning rates tend to hurt final performance in domains of higher action dimensionality.
E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS
For the ablation analysis, we use the same experimental setup as in the continuous control experiments while removing one component at a time.
To evaluate the effectiveness of Retrace/Q(λ) with off-policy correction, we replace both with importance sampling based estimates (following Degris et al. (2012)) which can be expressed recursively: Rt = rt + ρt+1Rt+1.
To evaluate the Stochastic Dueling Networks, we replace it with two separate networks: one computing the state values and the other Q values. Given Qret(xt, at), the naive way of estimating the state values is to use the following update rule:
( ρtQ ret(xt, at)− Vθv (xt) ) ∇θvVθv (xt).
The above update rule, however, suffers from high variance. We consider instead the following update rule:
ρt ( Qret(xt, at)− Vθv (xt) ) ∇θvVθv (xt)
which has markedly lower variance. We update our Q estimates as before.
To evaluate the effects of the truncation and bias correction trick, we change our c parameter (see Equation (16)) to∞ so as to use pure importance sampling.

This paper introduces an actor-critic deep RL approach with experience replay, which combines truncated importance sampling and trust region policy optimization. The paper also proposes a new method called stochastic duelling networks to estimate the critic for continuous action spaces. The method is applied to Atari games and continuous control problems, where it yields performance comparable to state-of-the-art methods.
As mentioned in the beginning of the paper, the main contributions of this work lies in combining 1) truncated importance sampling with retrace, 2) trust region policy optimization, and 3) stochastic duelling networks. These improvements work well and may be beneficial to future work in RL.
However, each improvement appears to be quite incremental. Moreover, the ACER framework seems much more complex and fragile to implement compared to the standard deep q-learning with prioritized replay (which appears to perform just as well on Atari games). So for the Atari domain, I would still put my money on prioritized replay due to its simplicity. Thirdly, improving sample efficiency for deep RL is a laudable goal, but really this goal should be pursued in a problem setting where sample efficiency is important. Unfortunately, the paper only evaluates sample efficiency in the Atari and continuous control tasks domain; two domains where sample efficiency is not important. Thus, it is not clear that the proposed method ACER will generalize to problems where we really care about sample efficiency.
Some technical aspects which need clarifications:
- For Retrace, I assume that you compute recursively $Q^{ret}$ starting from the end of each trajectory? Please comment on this.
- It's not clear to me how to derive eq. (7). Is an approximation (double tilde) sign missing?
- In section 3.1 the paper argued that $Q^{ret}$ gives a lower-variance estimate of the action-value function. Then why not use it in eq. (8) for the bias correction term?
- The paper states that it uses a replay memory of 50000 frames, so that across threads it is comparable in size to previous work. However, for each thread this is much smaller compared to earlier experiments on Atari games. For example, one million experience replay transitions were used in the paper "Prioritized Experience Replay" by Schaul et al. This may have a huge impact on performance of the models (both for ACER and for the competing models). In order to properly assess the improvements of ACER over previous work, the authors need to also experiment with larger experience replay memories.
Other comments:
- Please move Section 7 to the appendix.
- "Moreover, when using small values of lambda to reduce variance, occasional large importance weights can still cause instability": I think what is meant is using *large* values of lambda.
- Above eq. (6) mention that the squared error is used.
- Missing a "t" subscript at the beginning of eq. (9)?
- It was hard to understand the stochastic duelling networks. Please rephrase this part.
- Please clarify this sentence "To compare different agents, we adopt as our metric the median of the human normalized score over all 57 games."
- Figure 2 (Bottom): Please add label to vertical axes.


LEARNING APPROXIMATE DISTRIBUTION-SENSITIVE DATA STRUCTURES
We model representations as data-structures which are distribution sensitive, i.e., which exploit regularities in their usage patterns to reduce time or space complexity. We introduce probabilistic axiomatic specifications to extend abstract data structures which specify a class of representations with equivalent logical behavior to a distribution-sensitive data structures. We reformulate synthesis of distribution-sensitive data structures as a continuous function approximation problem, such that the functions of a data-structure deep neural networks, such as a stack, queue, natural number, set, and binary tree.
1 INTRODUCTION
Recent progress in artificial intelligence is driven by the ability to learn representations from data. Yet not all kinds of representations are equal, and many of the fundamental properties of representations (both as theoretical constructs and as observed experimentally in humans) are missing. Perhaps the most critical property of a system of representations is compositionality, which as described succinctly in (Fodor & Lepore, 2002), is when (i) it contains both primitive symbols and symbols that are complex; and (ii) the latter inherit their syntactic/semantic properties from the former. Compositionality is powerful because it enables a system of representation to support an infinite number of semantically distinct representations by means of combination. This argument has been supported experimentally; a growing body of evidence (Spelke & Kinzler, 2007) has shown that humans possess a small number of primitive systems of mental representation - of objects, agents, number and geometry - and new representations are built upon these core foundations.
Representations learned with modern machine learning methods possess few or none of these properties, which is a severe impediment. For illustration consider that navigation depends upon some representation of geometry, and yet recent advances such as end-to-end autonomous driving (Bojarski et al., 2016) side-step building explicit geometric representations of the world by learning to map directly from image inputs to motor commands. Any representation of geometry is implicit, and has the advantage that it is economical in only possessing information necessary for the task. However, this form of representation lacks (i) the ability to reuse these representations for other related tasks such as predicting object stability or performing mental rotation, (ii) the ability to compose these representations with others, for instance to represent a set or count of geometric objects, and (iii) the ability to perform explicit inference using representations, for instance to infer why a particular route would be faster or slower.
This contribution provides a computational model of mental representation which inherits the compositional and productivity advantages of symbolic representations, and the data-driven and economical advantages of representations learned using deep learning methods. To this end, we model mental representations as a form of data-structure, which by design possess various forms of compositionality. In addition, in step with deep learning methods we refrain from imposing a particular representations on a system and allow it instead be learned. That is, rather than specify a concrete data type (for example polygons or voxels for geometry), we instead define a class of representations as abstract data types, and impose invariants, or axioms, that any representation must adhere to.
Mathematicians have sought an axiomatic account of our mental representations since the end of the nineteenth century, but both as an account of human mental representations, and as a means of specifying representations for intelligent systems, the axiomatic specifications suffer from a number
of problems. Axioms are universally quantified - for all numbers, sets, points, etc - while humans, in contrast, are not uniformly good at manipulating numbers of different magnitude (Hyde, 2011; Nuerk & Willmes, 2005; Dehaene, 1997), rotating geometry of different shapes (Izard et al., 2011), or sets of different cardinality. Second, axioms have no algorithmic content; they are declarative rules which do not suggest how to construct concrete representations that satisfy them. Third, only simple systems have reasonable axioms, whereas many representations are complex and cannot in practice be fully axiomitized; conventional axiomatic specifications do not readily accommodate partial specification. A fourth, potentially fatal threat is offered by Dehaene (1997), where he shows that there are infinitely many systems, most easily dismissed by even a child as clearly not number-like, which satisfy Peano’s axioms of arithmetic. Moreover these ”nonstandard models of arithmetic” can never be eliminated by adding more axioms, leading Dehaene to conclude ”Hence, our brain does not rely on axioms.”.
We extend, rather than abandon, the axiomatic approach to specifying mental representations, and employ it purely as mechanism to embed domain specific knowledge. We model a mental representation as an implementation of an abstract data type which adheres approximately to a probabilistic axiomatic specification. We refer to this implementation as a distribution-sensitive data-structure.
In summary, in this paper:
• We introduce probabilistic axiomatic specifications as a quantifier-free relaxation of a conventional specification, which replaces universally quantified variables with random variables. • Synthesis of a representation is formulated as synthesis of functions which collectively
satisfy the axioms. When the axioms are probabilistic, this is amounts of maximizing the probability that the axiom is true. • We present a number of methods to approximate a probabilistic specification, reducing it
to a continuous loss function. • We employ neural networks as function approximators, and through gradient based opti-
mization learn representations for a number of fundamental data structures.
2 BACKGROUND: ABSTRACT DATA TYPES
Abstract data types model representations as a set of types and functions which act on values of those types. They can also be regarded as a generalized approach to algebraic structures, such as lattices, groups, and rings. The prototypical example of an abstract data type is the Stack, which models an ordered, first-in, last-out container of items. We can abstractly define a Stack of Items, in part, by defining the interface:
empty : Stack
push : Stack × Item→ Stack pop : Stack → Stack × Item
isempty : Stack → {0, 1}
The interface lists the function names and types (domains and range). Note that this is a functional (rather than imperative) abstract data type, and each function in the interface has no internal state. For example, push is a function that takes an instance of a Stack and an Item and returns a Stack. empty : Stack denotes a constant of type Stack, the empty stack of no items.
The meaning of the constants and functions is not specified in the interface. To give meaning to these names, we supplement the abstract data type with a specification as a set of axioms. The specification as a whole is the logical conjunction of this set of axioms. Continuing our example, for all s ∈ Stack, i ∈ Item:
pop(push(s, i)) = (s, i) (1) isempty(empty) = 1 (2)
isempty(push(s, i)) = 0 (3) pop(empty) = ⊥ (4)
A concrete representation of a stack is a data structure which assigns constants and functions to the names empty, push, pop and isempty. The data structure is a stack if and only if it satisfies the specification.
2.1 COMPOSITION OF DATA STRUCTURES
There are a number of distinct forms of compositionality with respect to data structures. One example is algorithmic compositionality, by which we can compose algorithms which use as primitive operations the interfaces to these representations. These algorithms can in turn form the interfaces to other representations, and so on.
An important property of an abstract data types which supports algorithmic compositionality is encapsulation. Encapsulation means that the particular details of how the functions are implemented should not matter to the user of the data type, only that it behaves as specified. Many languages enforce that the internals are unobservable, and that the data type can only be interacted with through its interface. Encapsulation means that data-structures can be composed without reasoning about their internal behavior.
In this paper however, we focus on parametric compositionality. Some data structures, in particular containers such as a stack, or set, or tree are parametric with respect to some other type, e.g. the type of item. Parametric compositionality means for example that if we have a representation of a set, and a representation of a number, we get a set of numbers for free. Or, given a representations for a tree and representations for Boolean logic, we acquire the ability to form logical expressions for free.
2.2 DISTRIBUTION SENSITIVE DATA STRUCTURES
Axiomatic specifications almost always contain universal quantifiers. The stack axioms are quantified over all possible stacks and all possible items. Real world use of a data structure is however never exhaustive, and rarely uniform. Continuing our stack example, we will never store an infinite number of items, and the distribution over how many items are stored, and in which order relative to each other, will highly non-uniform in typical use cases. Conventional data structures are agnostic to these distributional properties.
Data structures that exploit non-uniform query distributions are typically termed distributionsensitive (Bose et al., 2013), and are often motivated by practical concerns since queries observed in real-world applications are not uniformly random. An example is the optimum binary search tree on n keys, introduced by Knuth (Bose et al., 2013), which given a probability for each key has an average search cost no larger than any other key. More generally, distribution-sensitive data structures exploit underlying patterns in a sequence of operations in order to reduce time and space complexity.
3 PROBABILISTIC AXIOMATIC SPECIFICATION
To make the concept of a distribution-sensitive data-structure precise, we first develop the concept of an probabilistically axiomatized abstract data type (T,O, F ), which replaces universally quantified variables in its specification with random variables. T and O are respectively sets of type and interface names. F is a set of type specifications, each taking the form m : τ for a constant of type τ , or o : τ1 → τ2 denoting a function from τ1 to τ2. Here τ ∈ T or a Cartesian product T1×· · ·×Tn. A concrete data type σ implements an abstract data type by assigning a value (function or constant) to each name in O. A concrete data type is deemed a valid implementation only with respect to an algebraic specification A. A is a set of equational axioms of the form p = q, p and q are constants, random variables, or transformations of random variables by functions in O.
Since a transformation of a random variable yields a random variable, and an axiom is simply a predicate of its left and right hand side arguments, random variables present in an axiom implies that the axiom itself is a Boolean valued random variable. For example if we have a distribution over items i of the stack, axiom (1) itself is a random variable which is true or false depending on i, push, pop, and can only be satisfied with some probability. We let P [A(σ)] denote the probability
that a concrete data type σ satisfies the axioms:
P [A(σ)] := P [∧ipi = qi] (5)
Probabilistic axioms do not imply that the concrete data-structure itself is probabilistic. On the contrary, we are concerned with specifying and synthesizing deterministic concrete data structures which exploit uncertainty stemming only from the patterns in which the data-structure is used.
When P [A(σ)] = 1, σ can be said to fully satisfy the axioms. More generally, with respect to a space Σ of concrete data types, we denote the maximum likelihood σ∗ as one which maximizes the probability that the axioms hold:
σ∗ = arg max σ∈Σ P [A(σ)] (6)
4 APPROXIMATE ABSTRACT DATA TYPES
A probabilistic specification is not easier to satisfy than a universally quantified one, but it can lend itself more naturally to a number of approximations. In this section we outline a number of relaxations we apply to a probabilistic abstract data type to make synthesis tractable.
RESTRICT TYPES TO REAL VALUED ARRAYS
Each type τ ∈ T will correspond to a finite dimensional real valued multidimensional array Rn. Interface functions are continuous mappings between these arrays.
UNROLL AXIOMS
Axiom (1) of the stack is intensional in the sense that it refers to the underlying stack s. This provides an inductive property allowing us to fully describe the behavior of an unbounded number of push and pop operations with a single equational axiom. However, from an extensional perspective, we do not care about the internal properties of the stack; only that it behaves in the desired way. Put plainly, we only care that if we push an item i to the stack, then pop, that we get back i. We do not care that the stack is returned to its initial state, only that it is returned to some state that will continue to obey this desired behavior.
An extensional view leads more readily to approximation; since we cannot expect to implement a stack which satisfies the inductive property of axiom 1 if it is internally a finite dimensional vector. Instead we can unroll the axiom to be able to stack some finite number of n items:
APPROXIMATE DISTRIBUTIONS WITH DATA
We approximate random variables by a finite data distribution assumed to be a representative set of samples from that distribution. Given an axiom p = q, we denote p̂ and q̂ as values (arrays) computed by evaluating p and q respectively with concrete data from the data distributions of random variables and the interface functions.
RELAX EQUALITY TO DISTANCE
We relax equality constraints in axioms to a distance function, in particular the L2 norm. This transforms the equational axioms into a loss function. Given i axioms, the approximate maximum likelihood concrete data type σ̂∗ is then:
σ̂∗ = arg min i
∑ ‖p̂i − q̂i‖ (7)
Constants and parameterized functions (e.g. neural networks) which minimizes this loss function then compose a distribution-sensitive concrete data type.
5 EXPERIMENTS
We successfully synthesized approximate distribution-sensitive data-structures from a number of abstract data types:
• Natural number (from Peano’s axioms) • Stack • Queue • Set • Binary tree
With the exception of natural number (for which we used Peano’s axioms), we use axiomitizations from (Dale & Walker, 1996). As described in section 4, since we use finite dimensional representations we unroll the axioms some finite number of times (e.g., to learn a stack of three items rather than it be unbounded) and ”extensionalize” them.
In each example we used we used single layer convolutional neural networks with 24, 3 by 3 filters and rectifier non-linearities. In container examples such as Stack and Queue, the Item type was sampled from MNIST dataset, and the internal stack representation was chosen (for visualization) to also be a 28 by 28 matrix. We minimized the equational distance loss function described in section 3 using the adam optimization algorithm, with a learning rate of 0.0001 In figures 1 and 2 we visualize the properties of the learned stack.
To explore compositionality, we also learned a Stack,Queue and Set ofNumber, whereNumber was itself a data type learned from Peano’s axioms.
6 ANALYSIS
The learned internal representations depend on three things (i) the axioms themselves, (ii) the architecture of the networks for each function in the interface, and (iii) the optimization procedure. In the stack example, we observed that if we decreased the size of the internal representation of a stack, we would need to increase the size and complexity of the neural network to compensate. This implies that statistical information about images must be stored somewhere, but there is some flexibility over where.
Given the same architecture, the system learned different representations depending on the axioms and optimization. The stack representation learned in figure 1 differs from that in figure 3, indicating that there is not a unique solution to the problem, and different initialization strategies will yield different results. The queue internal representation is also different to them both, and the encoding is less clear. The queue and stack representations could have been the same (with only the interface functions push, pop, queue and dequeue taking different form).
As shown in figure 2, data-structures exhibit some generalization beyond the data distributions on which they are trained. In this case, a stack trained to store three items, is able to store four with some error, but degrades rapidly beyond that. Of course we cannot expect a finite capacity representation to store an unbounded number of items; lack of generalization is the cost of having optimized performance on the distribution of interest.
7 RELATED WORK
Our contribution builds upon the foundations of distribution-sensitive data structures (Bose et al., 2013), but departs from conventional work on distribution-sensitive data structures in that: (i) we
synthesize data structures automatically from specification, and (ii) the distributions of interest are complex data distributions, which prevents closed form solutions as in the optimum binary tree.
Various forms of machine learning and inference learn representations of data. Our approach bears resemblance to the auto-encoder (Bengio, 2009), which exploits statistics of a data distribution to learn a compressed representation as a hidden layer of a neural network. As in our approach, an auto-encoder is distribution sensitive by the constraints of the architecture and the training procedure (the hidden layer is of smaller capacity than the data and which forces the exploitation of regularities). However, an auto-encoder permits just two operations: encode and decode, and has no notion explicit notion of compositionality.
A step closer to our approach than the auto-encoder are distributed representations of words as developed in (Mikolov et al., 2000). These representations have a form of compositionality such that vector arithmetic on the representation results in plausible combinations (Air + Canada = AirCanada).
Our approach to learning representation can be viewed as a form of data-type synthesis from specification. From the very introduction of abstract data types, verification that a given implementation satisfies its specification was a motivating concern (Guttag et al., 1978; Guttag, 1978; Spitzen & Wegbreit, 1975). Modern forms of function synthesis (Solar-Lezama, 2009; Polikarpova & SolarLezama, 2016) use verification as an oracle to assist with synthesis. Our approach in a broad sense is similar, in that derivatives from loss function which is derived from relaxing the specification, guide the optimization through the paramterized function spaces.
Probabilistic assertions appear in first-order lifting (Poole, 2003), and Sampson (Sampson et al., 2014) introduce probabilistic assertions. Implementation of data type is a program. Main difference is that we synthesize data type from probabilistic assertion. Sumit’s work (Sankaranarayanan, 2014) seeks upper and lower bounds for the probability of the assertion for the programs which operate on uncertain data.
Recent work in deep learning has sought to embed discrete data structures into continuous form. Examples are the push down automata (Sun et al., 1993), networks containing stacks (Grefenstette et al., 2015), and memory networks (Sukhbaatar et al., 2015). Our approach can be used to synthesize arbitrary data-structure, purely from its specification, but is parameterized by the neural network structure. This permits it more generality, with a loss of efficiency.
8 DISCUSSION
In this contribution we presented a model of mental representations as distribution sensitive data structures, and a method which employs neural networks (or any parameterized function) to synthesize concrete data types from a relaxed specification. We demonstrated this on a number of examples, and visualized the results from the stack and queue.
One of the important properties of conventional data structures is that they compose; they can be combined to form more complex data structures. In this paper we explored a simple form of parametric composition by synthesizing containers of numbers. This extends naturally to containers of containers, .e.g sets of sets, or sets of sets of numbers. Future work is to extend this to richer forms of composition. In conventional programming languages, trees and sets are often made by composing arrays, which are indexed with numbers. This kind of composition ls fundamental to building complex software from simple parts.
In this work we learned representations from axioms. Humans, in contrast, learn representations mostly from experience in the world. One rich area of future work is to extend data-structure learning to the unsupervised setting, such that for example an agent operating in the real world would learn a geometric data-structures purely from observation.

The paper presents a framework to formulate data-structures in a learnable way. It is an interesting and novel approach that could generalize well to interesting datastructures and algorithms. In its current state (Revision of Dec. 9th), there are two strong weaknesses remaining: analysis of related work, and experimental evidence.
Reviewer 2 detailed some of the related work already, and especially DeepMind (which I am not affiliated with) presented some interesting and highly related results with its neural touring machine and following work. While it may be of course very hard to make direct comparisons in the experimental section due to complexity of the re-implementation, it would at least be very important to mention and compare to these works conceptually.
The experimental section shows mostly qualitative results, that do not (fully) conclusively treat the topic. Some suggestions for improvements:
* It would be highly interesting to learn about the accuracy of the stack and queue structures, for increasing numbers of elements to store.
* Can a queue / stack be used in arbitrary situations of push-pop operations occuring, even though it was only trained solely with consecutive pushes / consecutive pops? Does it in this enhanced setting `diverge' at some point?
* The encoded elements from MNIST, even though in a 28x28 (binary?) space, are elements of a ten-element set, and can hence be encoded a lot more efficiently just by `parsing' them, which CNNs can do quite well. Is the NN `just' learning to do that? If so, its performance can be expected to strongly degrade when having to learn to stack more than 28*28/4=196 numbers (in case of an optimal parser and loss-less encoding). To argue more in this direction, experiments would be needed with an increasing number of stack / queue elements. Experimenting with an MNIST parsing NN in front of the actual stack/queue network could help strengthening or falsifying the claim.
* The claims about `mental representations' have very little support throughout the paper. If indication for correspondence to mental models, etc., could be found, it would allow to hold the claim. Otherwise, I would remove it from the paper and focus on the NN aspects and maybe mention mental models as motivation.