We introduce a hybrid CPU/GPU version of the Asynchronous Advantage ActorCritic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU’s computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C.
1 INTRODUCTION
In the past, the need for task-specific, or even hand-crafted, features limited the application of Reinforcement Learning (RL) in real world problems (Sutton & Barto, 1998). However, the introduction of Deep Q-Learning Networks (DQN) (Mnih et al., 2015) revived the use of Deep Neural Networks (DNNs) as function approximators for value and policy functions, unleashing a rapid series of advancements. Remarkable results include learning to play video games from raw pixels (Bellemare et al., 2016; Lample & Singh Chaplot, 2016) and demonstrating super-human performance on the ancient board game Go (Silver et al., 2016). Research has yielded a variety of effective training formulations and DNN architectures (van Hasselt et al., 2015; Wang et al., 2015), as well as methods to increase parallelism while decreasing the computational cost and memory footprint (Nair et al., 2015; Mnih et al., 2016). In particular, Mnih et al. (2016) achieve state-of-the-art results on many gaming tasks through a novel lightweight, parallel method called Asynchronous Advantage ActorCritic (A3C). When the proper learning rate is used, A3C learns to play an Atari game (Brockman et al., 2016) from raw screen inputs more quickly and efficiently than previous methods: on a 16- core CPU, A3C achieves higher scores than previously published methods run for the same amount of time on a GPU.
Our study sets aside many of the learning aspects of recent work and instead delves into the computational issues of deep RL. Computational complexities are numerous, largely centering on a common factor: RL has an inherently sequential aspect, since the training data are generated while learning. The DNN model is constantly queried to guide the actions of agents whose gameplay in turn feeds DNN training. Training batches are commonly small and must be efficiently shepherded from the agents and simulator to the DNN trainer. When using a GPU, the mix of small DNN architectures, small training batch sizes, and contention for the GPU for both inference and training can lead to a severe under-utilization of the computational resources.
To systematically investigate these issues, we implement both CPU and GPU versions of A3C in TensorFlow (TF) (Abadi et al., 2015), optimizing each for efficient system utilization and to approximately replicate published scores in the Atari 2600 environment (Brockman et al., 2016). We
analyze a variety of “knobs” in the system and demonstrate effective automatic tuning of those during training. Our hybrid CPU/GPU implementation of A3C, named GA3C, generates and consumes training data substantially faster than its CPU counterpart, up to ∼ 6× faster for small DNNs and ∼ 45× for larger DNNs. While we focus on the A3C architecture, this analysis can be helpful for researchers and framework developers designing the next generation of deep RL methods.
2 RELATED WORK
Recent advances in deep RL have derived from both novel algorithmic approaches and related systems optimizations. Investigation of the algorithmic space seems to be the most common approach among researchers. Deep Q-Learning Networks (DQN) demonstrate a general approach to the learning problem (Mnih et al., 2015), relying heavily on the introduction of an experience replay memory to stabilize the learning procedure. This improves reliability but also increases the computational cost and memory footprint of the algorithm. Inspired by DQN, researchers have proposed more effective learning procedures, achieving faster and more stable convergence: Prioritized DQN (Schaul et al., 2015) makes better use of the replay memory by more frequently selecting frames associated with significant experiences. Double-DQN (van Hasselt et al., 2015) separates the estimate of the value function from the choice of actions (policy), thus reducing the tendency in DQN to be overly optimistic when evaluating its choices. Dueling Double DQN (Wang et al., 2015) goes a step further by explicitly splitting the computation of the value and advantage functions within the network. The presence of the replay memory makes the DQN approaches more suitable for a GPU implementation when compared to other LR methods, but state-of-the-art results are achieved by A3C (Mnih et al., 2016), which does not make use of it.
Among systems approaches, AlphaGo (Silver et al., 2016) recently achieved astonishing results through combined algorithmic and hardware specialization. The computational effort is impressive: 40 search threads, 1202 CPUs, and 176 GPUs are used in the distributed version for inference only. Supervised training took around three weeks for the policy network, using 50 GPUs, and another day using the RL approach for refinement. A similar amount of time was required to train the value network. Gorilla DQN (Nair et al., 2015) is a similarly impressive implementation of distributed RL system, achieving a significant improvement over DQN. The system requires 100 concurrent actors on 31 machines, 100 learners and a central parameter server with the network model. This work demonstrates the potential scalability of deep RL algorithms, achieving better results in less time, but with a significantly increased computational load, memory footprint, and cost.
3 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C)
3.1 REINFORCEMENT LEARNING BACKGROUND
In standard RL, an agent interacts with an environment over a number of discrete time steps. At each time step t, the agent observes a state st and, in the discrete case, selects an action at from the set of valid actions. An agent is guided by policy π, a function mapping from states st to actions at. After each action, the agent observes the next state st+1 and receives feedback in the form of a reward rt. This process continues until the agent reaches a terminal state or time limit, after which the environment is reset and a new episode is played.
The goal of learning is to find a policy π that maximizes the expected reward. In policy-based modelfree methods, a function approximator such as a neural network computes the policy π(at|st; θ), where θ is the set of parameters of the function. There are many methods for updating θ based on the rewards received from the environment. REINFORCE methods (Williams, 1992) use gradient ascent on E[Rt], where Rt = ∑∞ i=0 γ
irt+i is the accumulated reward starting from time step t and increasingly discounted at each subsequent step by factor γ ∈ (0, 1]. The standard REINFORCE method updates θ using the gradient ∇θ log π(at|st; θ)Rt, which is an unbiased estimator of ∇θ E[Rt]. The variance of the estimator is reduced by subtracting a learned baseline (a function of the state bt(st)) and using the gradient ∇θ log π(at|st; θ) ( Rt − bt(st)
) instead. One common baseline is the value function defined as V π(st) = E[Rt|st] which is the expected return for following the policy π in state st. In this approach the policy π and the baseline bt can be viewed as actor and critic in an actor-critic architecture (Sutton & Barto, 1998).
3.2 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C)
A3C (Mnih et al., 2016), which achieves state-of-the-art results on many gaming tasks including Atari 2600, uses a single DNN to approximate both the policy and value function. The DNN has two convolutional layers with 16×8×8 filters with a stride of 4, and 32×4×4 filters with a stride of 2, followed by a fully connected layer with 256 units; each hidden layer is followed by a rectifier nonlinearity. The two outputs are a softmax layer which approximates the policy function π (at|st; θ), and a linear layer to output an estimate of V (st; θ). Multiple agents play concurrently and optimize the DNN through asynchronous gradient descent. Similar to other asynchronous methods, the network weights are stored in a central parameter server (Figure 1a). Agents calculate gradients and send updates to the server after every tmax = 5 actions, or when a terminal state is reached. After each update, the central server propagates new weights to the agents to guarantee they share a common policy.
Two cost functions are associated with the two DNN outputs. For the policy function, this is:
fπ (θ) = log π (at|st; θ) (Rt − V (st; θt)) + βH (π (st; θ)) , (1) where θt are the values of the parameters θ at time t, Rt = ∑k−1 i=0 γ irt+i + γ kV (st+k; θt) is the estimated discounted reward in the time interval from t to t + k and k is upper-bounded by tmax, while H (π (st; θ)) is an entropy term, used to favor exploration during the training process. The factor β controls the strength of the entropy regularization term. The cost function for the estimated value function is:
fv (θ) = (Rt − V (st; θ))2 . (2)
Training is performed by collecting the gradients ∇θ from both of the cost functions and using the standard non-centered RMSProp algorithm (Tieleman & Hinton, 2012) as optimization:
g = αg + (1− α)∆θ2 θ ← θ − η∆θ/ √ g + .
(3)
The gradients g can be either shared or separated between agent threads but the shared implementation is known to be more robust (Mnih et al., 2016).
The original implementation of A3C (Mnih et al., 2016) uses 16 agents on a 16 core CPU and it takes about four days to learn how to play an Atari game (Brockman et al., 2016). The main reason for using CPU other than GPU, is the inherently sequential nature of RL in general, and A3C in particular. In RL, the training data are generated while learning, which means the training and inference batches are small and GPU is mostly idle during the training, waiting for new data to arrive. Since A3C does not utilize any replay memory, it is completely sequential and therefore a CPU implementation is as fast as a naive GPU implementation.
4 HYBRID CPU/GPU A3C (GA3C)
We propose GA3C, an alternative architecture of A3C, with emphasize on an efficient GPU utilization to increase the number of training data generated and processed per second. We demonstrate that our implementation of GA3C effectively converges significantly faster than our CPU implementation of A3C, achieving the state-of-the-art performance in a shorter time.
4.1 GA3C ARCHITECTURE
The primary components of GA3C (Figure 1b) are a DNN with training and prediction on a GPU, as well as a multi-process, multi-thread CPU architecture with the following components:
• Agent is a process interacting with the simulation environment: choosing actions according to the learned policy and gathering experiences for further training. Similar to A3C, multiple concurrent agents run independent instances of the environment. Unlike the original, each agent does not have its own copy of the model. Instead it queues policy requests in a Prediction Queue before each action, and periodically submits a batch of input/reward experiences to a Training Queue; the size of each training batch is typically equal to tmax experiences, though it is sometimes smaller for experiences collected at the end of an episode.
• Predictor is a thread which dequeues as many prediction requests as are immediately available and batches them into a single inference query to the DNN model on the GPU. When predictions are completed, the predictor returns the requested policy to each respective waiting agent. To hide latency, one or more predictors can act concurrently.
• Trainer is a thread which dequeues training batches submitted by agents and submits them to the GPU for model updates. GPU utilization can be increased by grouping training batches among several agents; we found that this generally leads to a more stable convergence, but the convergence speed is reduced when the merged training batches are too large; a compromise is explored in Section 5.3. Multiple trainers may run in parallel to hide latency.
Unlike A3C, GA3C maintains only one copy of the DNN model (Fig. 1a and 1b), centralizing predictions and training updates and removing the need for synchronization. Also, in comparison with A3C, agents in GA3C do not compute the gradients themselves. Instead, they send experiences to trainers that update the network on the GPU accordingly. This introduces a potential lag between the generation and consumption of experiences, which we analyze in detail in Sections 4.4 and 5.3.
4.2 PERFORMANCE METRICS AND TRADE-OFFS
The GA3C architecture exposes numerous tradeoffs for tuning its computational efficiency. In general, it is most efficient to transfer data to a GPU in large enough blocks to maximize the usage of the bandwidth between the GPU and CPU. Application performance on the GPU is optimized when the application has large amounts of parallel computations that can hide the latency of fetching data from memory. Thus, we want to maximize the parallel computations the GPU is performing, maximize the size of data transfer to the GPU, and minimize the number of transfers to the GPU. Increasing the number of predictors, NP , allows faster fetching prediction queries, but leads to smaller prediction batches, resulting in multiple data transfers and overall lower GPU utilization. A larger number of trainers, NT , potentially leads to more frequent updates to the model, but an overhead is paid when too many trainers occupy the GPU while predictors cannot access it. Lastly, increasing the number of agents, NA, ideally generates more training experiences while hiding prediction latency. However, we would expect diminishing returns from unnecessary context switching overheads after exceeding some threshold depending on the number of CPU cores.
These aspects are well captured by a metric like the Trainings Per Second (TPS), which is the rate at which we remove batches from the training queue. It corresponds to the rate of model updates and it is approximately proportional to the overall learning speed, given a fixed learning rate and training batch size. Another metric is the Predictions Per Second (PPS), the rate of issuing prediction queries from prediction queue, which maps to the combined rate of gameplay among all agents. Notice that in A3C a model update occurs every time an agent plays tmax = 5 actions (Mnih et al., 2016).
Hence, in a balanced configuration, PPS≈TPS×tmax. Since each action is repeated four times as in (Mnih et al., 2016), the number of frames per second is 4×PPS. Computational aspects are not disconnected from the convergence of the learning algorithm. For instance, employing too many agents will tend to fill the training queue, introducing a significant time delay between agent experiences (at, st and Rt in Eq. (1)) and the corresponding model updates, possibly threatening model convergence (see Section 4.4). Another example is batching of training data: larger batches improves GPU occupancy by increasing the parallelism. They also decrease the TPS (i.e., the number of model updates per second), increasing the chance that the DNN model used in prediction and to compute the gradient in Eq. (4) are indeed the same model. The consequence (experimentally observed, see Section 5.3) is an increased stability of the learning process but, beyond a certain training batch size, this leads to a reduction in the convergence speed. In short, NT , NP , and NA encapsulate many complex dynamics relating both computational and convergence aspects of the learning procedure. Their effect on the convergence of the learning process has to be measured by analyzing not only TPS but also the learning curves.
4.3 DYNAMIC ADJUSTMENT OF TRADE-OFFS
The setting of NP , NT and NA that maximizes the TPS depends on many aspects such as the computational load of the simulation environment, the size of the DNN, and the available hardware. As a rule of thumb, we found that the number of agents NA should at least match the available CPU cores, with two predictors and two trainers NP = NT = 2. However, this rule hardly generalizes to a large variety of different situations and only occasionally corresponds to the computationally most efficient configuration. Therefore, we propose an annealing process to configure the system dynamically. Every minute, we randomly change NP , NT , or NA by ±1, monitoring alterations in TPS to accept or reject the new setting. The optimal configuration is then automatically identified in a reasonable time, for different environments or systems. Figure 2 shows the automatic adjustment procedure finding two different optimal settings for two different games, on the same real system.
4.4 POLICY LAG IN GA3C
At a first sight, GA3C and A3C are different implementations of the same algorithm, but GA3C has a subtle difference which affects the stability of the algorithm. This problem is caused by the latency between the time t − k, when a training example has been generated, and when it is consumed for training, t, essentially changing the gradients to:
∇θ [ log π (at−k|st−k; θ) (Rt−k − V (st−k; θt)) + βH (π (st−k; θ)) ] . (4)
Since the Training Queue is not blocking, the states it contains can be old. The value of the delay k is bounded by the maximum size of the queue and influenced by how the system configuration balances training and prediction rates. In other words, the DNN controller selects the action at−k at time t− k; the corresponding experience lies in a training queue until time t, when a trainer thread pops the element out of the queue to compute the gradient as in Eq. (4). The DNN controller at time t generally differs from the one at time t − k, since trainers can modify the DNN weights at any time. Therefore, the policy and value function π and V used to compute the gradient at time t will
differ from those used at time t − k to collect the experience, whereas the action used to compute the gradient in Eq. (4) remains at−k.
This delay can lead to instabilities for two reasons. The first one is the possible generation of very large values in log π (at−k|st−k; θt). In fact, π (at−k|st−k; θt−k) is generally large, since it is the probability of sampled action at−k, but over the course of lag k new parameters θt can make π (at−k|st−k; θt) very small. In the worst case, the updated probability is zero, generating infinite values in the log and causing optimization to fail. To avoid this, we add a small term > 0:
∇θ [ log ( π (at−k|st−k; θ) + ) (Rt−k − V (st−k; θt)) + βH (π (st−k; θ) + ) ] . (5)
Beyond fixing the error in the case π = 0, this fix also improves the stability of the algorithm and removes the necessity of gradient clipping. In fact, as ∂ log(π + )/∂θ = (∂π/∂θ)/(π + ), establishes an upper bound for the multiplicative factor in front of ∂π/∂θ. A similar term is also added in the entropy computation to avoid a similar explosion. It is important to remember that even A3C suffers from a similar issue. In fact, as the action at in Eq. (1) is selected by sampling the output softmax, there is a chance that π(at) is very small and therefore ∂ log(π)/∂θ is large. However, gradient clipping prevents the usage of a gradient with large magnitude in A3C.
The second reason for introducing instabilities in GA3C is a generalization of the first one. Since training and predictions are computed with potentially different DNN parameters, the resulting gradient is noisy, and can therefore lead to unreliable updates of the DNN weights. This is different from A3C, where every agent has its own copy of the model and uses it to compute both π and ∂ log(π)/∂θ, before synchronizing the DNN model with the other agents.
5 ANALYSIS
We profile the performance of GA3C and in the process seek to better understand the system dynamics of deep RL training on hybrid CPU/GPU systems. Experiments are conducted on the GPUenabled systems described in Table 1 and monitored with CUDA profilers and custom profiling code based on performance counter timing within Python. We present profiling and convergence experiments both with and without automatic adjustment of the number of agents NA, trainers NT , and predictors NP , and without constraints on the size of the prediction and training queues.
5.1 EFFECT OF RESOURCE UTILIZATION ON TPS
Maximizing training speed. To begin, consider raw training speed as expressed in model update frequency, or trainings per second (TPS). Figure 3 shows TPS on System I in Table 1 for the first 16 minutes of training on PONG. We consider numbers of agents NA ∈ {16, 32, 64, 128} and plot the top 3 combinations of NP , NT ∈{1, 2, 4, 8, 16}. On this system, increasing NA yields a higher TPS
up to NA= 128 where diminishing returns are observed, likely due to additional process overhead. The highest consistent TPS on this system is observed with NA = 128 and NP = NT = 2 with a speed-up of ∼ 4× relative to the CPU-only implementation (see Table 2).
GPU utilization and DNN size. The fastest configuration (NA = 128, NP =NT =2) for System I in Table 1 has an average GPU utilization time of only 56%, with average and peak occupancy of 76% and 98%, respectively.1 This suggests there is computational capacity for a larger network model. Therefore we profile GA3C on a series of deeper DNN architectures2 to evaluate this hypothesis. Figure 3 shows that TPS drops by only 7% with a one-layer deeper DNN controller; at the same time, the
average GPU utilization and occupancy increase by approximately 12% and 0.5%, respectively. The 7% drop in TPS is the consequence of the increased depth which forces an additional serial computational on the GPU (and therefore a 12% increase in its utilization). The negligible 0.5% increase in occupancy is likely explained by an efficient management of the computational resources by cuDNN; there is still room available to run additional parallel tasks (or, in other words, a wider DNN) at minimal cost.
1Occupancy reflects the parallelism of the computation by capturing the proportion of GPU threads which are active while the GPU is in use. For example, occupancy of 76% with utilization of 56% implies roughly 43% overall thread utilization.
2We increase the number of filters in the first layer of the DNN from 16 to 32, and add a third convolutional layer with 64 4 × 4 filters and stride 2. To generate four large DNNs with varying computational cost, we reduce the stride on the first layer from 4 pixels to 1 pixel, with a step of 1 pixel. Note the DNN with stride 1 in the first layer also has 4× larger fully-connected layer.
By reducing the stride of the first layer of the DNN (in addition to adding a convolutional layer), we scale DNN size with finer granularity, and we compare FPS between GA3C and our CPU implementation of A3C. Table 2 shows the speed up provided by GA3C increases as the DNN grows. This is mainly due to increasing GPU utilization, as reported in Table 2. With the largest DNN, our CPU implementation achieves TPS ≈ 11, which is approximately 45× slower than GA3C. This behavior is consistent across different systems, as shown in Table 2, where the CPU implementation of A3C using the largest DNN with stride 1 is 7× (System III) to 32× (System I) slower than the small network. Scaling with DNN size is more favorable on a GPU, with a slow down factor of 4.9× in the worst case (System III) and 2.2× in the best case (System II). Further, more recent GPUs (Maxwell architecture) scale better (2.2× and 2.7× slow down for Systems I and II) than older GPUs (4.8× slow down for the Kepler architecture, System III).
Generally speaking, for large DNNs, maximum TPS and FPS are achieved by intensively using the GPU for prediction and training, while the CPU runs the simulation environment and remains mostly idle. In practice, this allows experimenting with larger architectures, which may be particularly important for real world problems, e.g. robotics or autonomous driving (Lillicrap et al., 2015). Moreover, the idle CPU represents an additional computational resource, but such investigation is beyond the scope of this paper.
Significant latency. Profiling on System I in Table 1 reveals that the average time spent by an agent waiting for a prediction call to be completed is 108ms, only 10% of which is taken by the GPU inference. The remaining 90% is overhead spent accumulating the batch and calling the prediction function in Python. Similarly, for training we find that of the average 11.1ms spent performing a DNN update, 59% is overhead. This seems to suggest that a more optimized implementation (possibly based on a low level language like C++) may reduce these overheads, but this investigation remains for future work.
Manually balancing components. Agents, predictors, and trainers all share the GPU as a resource; thus balance is important. Figure 3 shows the top three performing configurations of NP and NT for different numbers of agents, NA, with System I in Table 1. A 14% drop in TPS is exhibited between the best and worst depicted configuration, despite the exclusion of all but the top three performers for each number of agents. The best results have 4 or fewer predictor threads, seemingly preventing batches from becoming too small. The NP : NT ratios for top performers tend to be 1 : 2, 1 : 1, or 2 : 1, whereas higher ratios such as 1 : 8 and 1 : 4 are rarely successful, likely due to the implicit dependence of training on prediction speed. However, if the training queue is too full, training calls take more GPU time, thereby throttling prediction speed. This is further confirmed by our experimental finding that TPS and PPS plots track closely. Figure 4 shows training queue size and prediction batch size for the top configurations. In all cases, the training queue stabilizes well below its maximum capacity. Additionally, the fastest configuration has one of the largest average prediction batch sizes, yielding higher GPU utilization.
5.2 EFFECT OF TPS ON LEARNING SPEED
The beneficial effect of an efficient configuration on the training speed is shown in Figure 5. Training with a suboptimal configuration (e.g. NP = NT = NA = 1 or NP = NT = 1, NA = 16) leads to a severe underutilization of the GPU, a low TPS, and a slow training process. Using the optimal configuration achieves a much higher score in a shorter period of time, mainly driven by playing more frames, i.e. collecting more experiences, in the same amount of time.
Mnih et al. (2016) note that asynchronous methods generally achieve significant speedups from using a greater number of agents, and even report superlinear speedups for asynchronous one-step Q-learning. It is worth noting that optimal configurations for GA3C generally employ a much higher number of agents compared to the CPU counterpart, e.g. the optimal configuration for System I in Table 1 uses 128 agents. This suggests that GPU implementations of asynchronous learning methods may benefit from both a higher TPS and from collecting experiences from a wider number of agents.
The learning curve for GA3C with dynamic configuration (Figure 5) tracks closely with the learning curve of the optimal configuration. The total number of frames played is generally slightly lower over the same time due to the search procedure overhead: the configuration is changed once every minute, tending to oscillate around the optimal configuration. Notice also that, in Figure 5, the starting point of the dynamic configuration is NT = NP = NA = 1, which is much slower than the optimal configuration. But scoring performance is nearly identical, indicating that the dynamic method may ease the burden of configuring GA3C on a new system.
Table 3 compares scores achieved by A3C on the CPU (as reported in (Mnih et al., 2016)) with the best agent trained by our TensorFlow implementation of GA3C. Unfortunately, a direct speed comparison is infeasible without either the original source code or the average number of frames or training updates per second. However, results in this table do show that after one day of training our open-source implementation can achieve similar scores to A3C after four days of training.
Figure 6 shows typical training curves for GA3C on several Atari games as a function of wallclock time. When compared to the training curves reported in Mnih et al. (2016), GA3C shows faster convergence toward the maximum score in a shorter time for certain games such as PONG, convergence towards a better score in a larger amount of time (e.g. QBERT) or, for other games, a slower convergence rate (e.g. BREAKOUT). It has to be noted, however, that data reported by Mnih et al. (2016) are the average learning curves of the top five learners in a set of fifty learners, each with a different learning rate. On the other hand, in Figure 6, we are reporting three different runs for two (not essentially optimal) learning rates, fixed for all the games. This demonstrates some robustness of GA3C with respect to the choice of the learning rate, whereas it is also likely that better learning curves can be obtained using optimized learning rates. A deeper investigation on a large amount of data, potentially facilitated by our release of the GA3C code, may also reveal how peculiarities of each game differently affect the convergence of A3C and GA3C, but this goes beyond the scope of this paper.
5.3 POLICY LAG, LEARNING STABILITY AND CONVERGENCE SPEED
One of the main differences between GA3C and A3C is the asynchronous computation of the forward step (policy π) and the gradients (Eq. (4)) used to update the DNN. Delays between these
two operations may introduce noise in the gradients, making the learning process unstable. We experimentally investigated the impact of this asynchrony on the learning process to determine if a synchronization mechanism, which may negatively impact both PPS and TPS, can increase the stability of the algorithm.
In GA3C, each agent generally pushes tmax experiences in the training queue. By default, trainers collect a batch of experiences from a single agent from the training queue and send the batch to the GPU to compute gradients, as in Eq. (5). Each time a trainer updates the DNN weights, the remaining experiences in the training queue are no longer in sync with the DNN model. This situation becomes worse when the average length of the training queue is large.
By allowing larger training batch sizes, we reduce the number of DNN updates per second (TPS), and consequently diminish the effect of the delay k in Eq. (5). In this way we increase the chance that the collected experiences and the computed gradients are in sync, which improves the stability. Notice that, even if the TPS is lower, the average magnitude of the updates is indeed larger, since we sum the gradients computed over the training batch.
In this setting, the optimal training batch size compromises among TPS, the average gradient step magnitude and the training stability. Another factor to be considered is that batching training data potentially leverages the GPU computational capability better by reducing the time devoted to compute the DNN updates while increasing the GPU occupancy during this phase. This gives more GPU time to the predictors, potentially increasing the PPS. However, this advantage tends to disappear when the training batch size is too large and predictors stay idle while the DNN update is computed.
Figure 7 compares convergence curves when no minimum size for training batch is compulsory (the default GA3C implementation where gradient updates are computed on a single agent’s batch) and when a minimum training batch size is enforced (combining multiple agent batches into a single gradient update). In the latter case, trainers collect experiences from multiple agents at the same time from the training queue and send them to the GPU for computation of gradients as in Eq. (5). Up to a certain batch size (between 20 and 40, in our experiments), increasing the training batch size stabilizes the learning procedure and generally leads to faster convergence. Some games such as PONG indeed do not suffer from this instability, and the effect of the minimum batch size is less evident in this case. We speculate that a careful selection of the learning rate combined with the proper minimum training batch size may lead to even faster convergence.
6 CONCLUSION
By investigating the computational aspects of our hybrid CPU/GPU implementation of GA3C, we achieve a significant speed up with respect to its CPU counter part. This comes as a result of a flexible system capable of finding a reasonable allocation of the available computational resources. Our approach allows producing and consuming training data at the maximum pace on different systems, or to adapt to temporal changes of the computational load on one system. Despite the fact that we analyze A3C only, most of our findings can be applied to similar RL asynchronous algorithms.
We believe that the analysis of the computational aspects of RL algorithms may be a consistent theme in RL in the future, motivating further studies such as this one. The potential benefits of such investigation goes well beyond the computational aspects. For instance, we demonstrate that GA3C scales with the size of the DNN much more efficiently than our CPU implementation of A3C, thus opening the possibility to explore the use of large DNN controllers to solve real world RL problems.
By open sourcing GA3C (see https://github.com/NVlabs/GA3C), we allow other researchers to further explore this space, investigate in detail the computational aspects of deep RL algorithms, and test new algorithmic solutions, including strategies for the combined utilization of the CPU and GPU computational resources.
ACKNOWLEDGMENTS
We thank Prof. Roy H. Campbell for partially supporting this work.

This paper introduces a 'GPU-friendly' variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive.
One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn't provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn't have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made.
I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial.
A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.


Learning a natural language interface for database tables is a challenging task that involves deep language understanding and multi-step reasoning. The task is often approached by mapping natural language queries to logical forms or programs that provide the desired response when executed on the database. To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce such programs on a real-world dataset. We enhance the objective function of Neural Programmer, a neural network with built-in discrete operations, and apply it on WikiTableQuestions, a natural language question-answering dataset. The model is trained end-to-end with weak supervision of question-answer pairs, and does not require domain-specific grammars, rules, or annotations that are key elements in previous approaches to program induction. The main experimental result in this paper is that a single Neural Programmer model achieves 34.2% accuracy using only 10,000 examples with weak supervision. An ensemble of 15 models, with a trivial combination technique, achieves 37.7% accuracy, which is competitive to the current state-of-the-art accuracy of 37.1% obtained by a traditional natural language semantic parser. 1 BACKGROUND AND INTRODUCTION Databases are a pervasive way to store and access knowledge. However, it is not straightforward for users to interact with databases since it often requires programming skills and knowledge about database schemas. Overcoming this difficulty by allowing users to communicate with databases via natural language is an active research area. The common approach to this task is by semantic parsing, which is the process of mapping natural language to symbolic representations of meaning. In this context, semantic parsing yields logical forms or programs that provide the desired response when executed on the databases (Zelle & Mooney, 1996). Semantic parsing is a challenging problem that involves deep language understanding and reasoning with discrete operations such as counting and row selection (Liang, 2016). The first learning methods for semantic parsing require expensive annotation of question-program pairs (Zelle & Mooney, 1996; Zettlemoyer & Collins, 2005). This annotation process is no longer necessary in the current state-of-the-art semantic parsers that are trained using only question-answer pairs (Liang et al., 2011; Kwiatkowski et al., 2013; Krishnamurthy & Kollar, 2013; Pasupat & Liang, 2015). However, the performance of these methods still heavily depends on domain-specific grammar or pruning strategies to ease program search. For example, in a recent work on building semantic parsers for various domains, the authors hand-engineer a separate grammar for each domain (Wang et al., 2015). Recently, many neural network models have been developed for program induction (Andreas et al., 2016; Jia & Liang, 2016; Reed & Freitas, 2016; Zaremba et al., 2016; Yin et al., 2015), despite ∗Work done at Google Brain.
1 BACKGROUND AND INTRODUCTION
Databases are a pervasive way to store and access knowledge. However, it is not straightforward for users to interact with databases since it often requires programming skills and knowledge about database schemas. Overcoming this difficulty by allowing users to communicate with databases via natural language is an active research area. The common approach to this task is by semantic parsing, which is the process of mapping natural language to symbolic representations of meaning. In this context, semantic parsing yields logical forms or programs that provide the desired response when executed on the databases (Zelle & Mooney, 1996). Semantic parsing is a challenging problem that involves deep language understanding and reasoning with discrete operations such as counting and row selection (Liang, 2016).
The first learning methods for semantic parsing require expensive annotation of question-program pairs (Zelle & Mooney, 1996; Zettlemoyer & Collins, 2005). This annotation process is no longer necessary in the current state-of-the-art semantic parsers that are trained using only question-answer pairs (Liang et al., 2011; Kwiatkowski et al., 2013; Krishnamurthy & Kollar, 2013; Pasupat & Liang, 2015). However, the performance of these methods still heavily depends on domain-specific grammar or pruning strategies to ease program search. For example, in a recent work on building semantic parsers for various domains, the authors hand-engineer a separate grammar for each domain (Wang et al., 2015).
Recently, many neural network models have been developed for program induction (Andreas et al., 2016; Jia & Liang, 2016; Reed & Freitas, 2016; Zaremba et al., 2016; Yin et al., 2015), despite
∗Work done at Google Brain.
the notorious difficulty of handling discrete operations in neural networks (Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016). Most of these approaches rely on complete programs as supervision (Jia & Liang, 2016; Reed & Freitas, 2016) while others (Zaremba et al., 2016; Yin et al., 2015) have been tried only on synthetic tasks. The work that is most similar to ours is that of Andreas et al. (2016) on the dynamic neural module network. However, in their method, the neural network is employed only to search over a small set of candidate layouts provided by the syntactic parse of the question, and is trained using the REINFORCE algorithm (Williams, 1992). Hence, their method cannot recover from parser errors, and it is not trivial to adapt the parser to the task at hand. Additionally, all their modules or operations are parametrized by a neural network, so it is difficult to apply their method on tasks that require discrete arithmetic operations. Finally, their experiments concern a simpler dataset that requires fewer operations, and therefore a smaller search space, than WikiTableQuestions which we consider in our work. We discuss other related work in Section 4.
Neural Programmer (Neelakantan et al., 2016) is a neural network augmented with a set of discrete operations. It produces both a program, made up of those operations, and the result of running the program against a given table. The operations make use of three variables: row selector, scalar answer, and lookup answer, which are updated at every timestep. lookup answer and scalar answer store answers while row selector is used to propagate information across time steps. As input, a model receives a question along with a table (Figure 1). The model runs for a fixed number of time steps, selecting an operation and a column from the table as the argument to the operation at each time step. During training, soft selection (Bahdanau et al., 2014) is performed so that the model can be trained end-to-end using backpropagation. This approach allows Neural Programmer to explore the search space with better sample complexity than hard selection with the REINFORCE algorithm (Williams, 1992) would provide. All the parameters of the model are learned from a weak supervision signal that consists of only the final answer; the underlying program, which consists of a sequence of operations and of selected columns, is latent.
In this work, we develop an approach to semantic parsing based on Neural Programmer. We show how to learn a natural language interface for answering questions using database tables, thus integrating differentiable operations that are typical of neural networks with the declarative knowledge contained in the tables and with discrete operations on tables and entries. For this purpose, we make several improvements and adjustments to Neural Programmer, in particular adapting its objective function to make it more broadly applicable.
In earlier work, Neural Programmer is applied only on a synthetic dataset. In that dataset, when the expected answer is an entry in the given table, its position is explicitly marked in the table. However, real-world datasets certainly do not include those markers, and lead to many ambiguities (e.g., (Pasupat & Liang, 2015)). In particular, when the answer is a number that occurs literally in the table, it is not known, a priori, whether the answer should be generated by an operation or selected from the table. Similarly, when the answer is a natural language phrase that occurs in multiple positions in the table, it is not known which entry (or entries) in the table is actually responsible for the answer. We extend Neural Programmer to handle the weaker supervision signal by backpropagating through decisions that concern how the answer is generated when there is an ambiguity.
Our main experimental results concern WikiTableQuestions (Pasupat & Liang, 2015), a real-world question-answering dataset on database tables, with only 10,000 examples for weak supervision. This dataset is particularly challenging because of its small size and the lack of strong supervision, and also because the tables provided at test time are never seen during training, so learning requires adaptation at test time to unseen column names. A state-of-the-art, traditional semantic parser that relies on pruning strategies to ease program search achieves 37.1% accuracy. Standard neural network models like sequence-to-sequence and pointer networks do not appear to be promising for this dataset, as confirmed in our experiments below, which yield single-digit accuracies. In comparison, a single Neural Programmer model using minimal text pre-processing, and trained end-to-end, achieves 34.2% accuracy. This surprising result is enabled primarily by the sample efficiency of Neural Programmer, by the enhanced objective function, and by reducing overfitting via strong regularization with dropout (Srivastava et al., 2014; Iyyer et al., 2015; Gal & Ghahramani, 2016) and weight decay. An ensemble of 15 models, even with a trivial combination technique, achieves 37.7% accuracy.
2 NEURAL PROGRAMMER
In this section we describe in greater detail the Neural Programmer model and the modifications we made to the model. Neural Programmer is a neural network augmented with a set of discrete operations. The model consists of four modules:
• Question RNN that processes the question and converts the tokens to a distributed representation. We use an LSTM network (Hochreiter & Schmidhuber, 1997) as the question RNN.
• A list of discrete operations such as counting and entry selection that are manually defined. Each operation is parameterized by a real-valued vector that is learned during training.
• A selector module that induces two probability distributions at every time step, one over the set of operations and another over the set of columns. The input to the selector is obtained by concatenating the last hidden state of the question RNN, the hidden state of the history RNN from the current timestep, and the attention vector obtained by performing soft attention (Bahdanau et al., 2014) on the question using the history vector. Following Neelakantan et al. (2016), we employ hard selection at test time.
• History RNN modeled by a simple RNN (Werbos, 1990) with tanh activations which remembers the previous operations and columns selected by the model. The input to the history RNN at each timestep is the result of concatenating the weighted representations of operations and columns with their corresponding probability distributions produced by the selector at the previous timestep.
A more detailed description of the basic model can be found in Neelakantan et al. (2016). The model runs for fixed total of T timesteps. The parameters of the operations, selector module, question and
history RNNs are all learned with backpropagation using a weak supervision signal that consists of the final answer. Below, we discuss several modifications to the model to make it more broadly applicable, and easier to train.
2.1 OPERATIONS
We use 15 operations in the model that were chosen to closely match the set of operations used in the baseline model (Pasupat & Liang, 2015). All the operations except select and most frequent entry operate only on the set of selected rows which is given by the row selector variable. Before the first timestep, all the rows in the table are set to be selected. The built-in operations are:
• count returns the number of selected rows in row selector. • select and most frequent entry are operations which are computed only once for every
question and output a boolean tensor with size same as the size of the input table. An entry in the output of the select operation is set to 1 if the entry matches some phrase in the question. The matched phrases in the question are anonymized to prevent overfitting. Similarly, for most frequent entry, it is set to 1 if the entry is the most frequently occurring one in its column.
• argmax, argmin, greater than, less than, greater than or equal to, less than or equal to are all operations that output a tensor with size same as the size of the input table.
• first, last, previous and next modify the row selector. • print operation assigns row selector on the selected column of lookup answer. • reset resets row selector to its initial value. This operation also serves as no-op when the
model needs to induce programs whose complexity is less than T .
All the operations are defined to work with soft selection so that the model can be trained with backpropagation. The operations along with their definitions are discussed in the Appendix.
2.2 OUTPUT AND ROW SELECTOR
Neural programmer makes use of three variables: row selector, scalar answer and lookup answer which are updated at every timestep. The variable lookup answer stores answers that are selected from the table while scalar answer stores numeric answers that are not provided in the table.1 The induced program transfers information across timesteps using the row selector variable which contains rows that are selected by the model.
Given an input table Π, containing M rows and C columns (M and C can vary across examples), the output variables at timestep t are given by:
scalar answert = α op t (count)outputt(count),
lookup answert [i][j] = α col t (j)α op t (print)row selectt−1 [i],∀(i, j)i = 1, 2, . . . ,M, j = 1, 2, . . . , C
where αopt (op) and α col t (j) are the probabilities assigned by the selector to operation op and column j at timestep t respectively and outputt(count) is the output of the count operation at timestep t. The row selector variable at timestep t is obtained by taking the weighted average of the outputs of the remaining operations and is discussed in the Appendix. lookup answerT [i][j] is the probability that the element (i, j) in the input table is in the final answer predicted by the model.
2.3 TRAINING OBJECTIVE
We modify the training objective of Neural Programmer to handle the supervision signal available in real-world settings. In previous work, the position of the answers are explicitly marked in the table when the answer is an entry from the table. However, as discussed in Section 1, in real-world datasets (e.g., (Pasupat & Liang, 2015)) the answer is simply written down introducing two kinds of ambiguities. First, when the answer is a number and if the number is in the table, it is not known
1It is possible to extend the model to generate natural language responses using an RNN decoder but it is not the focus of this paper and we leave it for further work.
whether the loss should be computed using the scalar answer variable or the lookup answer variable. Second, when the answer is a natural language phrase and if the phrase occurs in multiple positions in the table, we again do not know which entry (or entries) in the table is actually responsible for generating the answer. We extend Neural Programmer to handle this weaker supervision signal during training by computing the loss only on the prediction that is closest to the desired response.
For scalar answers we compute the square loss:
Lscalar(scalar answerT , y) = 1
2 (scalar answerT − y)2
where y is the ground truth answer. We divide Lscalar by the number of rows in the input table and do not backpropagate on examples for which the loss is greater than a threshold since it leads to instabilities in training.
When the answer is a list of items y = (a1, a2, . . . , aN ), for each element in the list (ai, i = 1, 2, . . . , N ) we compute all the entries in the table that match that element, given by Si = {(r, c), ∀ (r, c) Π[r][c] = ai}. We tackle the ambiguity introduced when an answer item occurs at multiple entries in the table by computing the loss only on the entry which is assigned the highest probability by the model. We construct g ∈ {0, 1}M×C , where g[i, j] indicates whether the element (i, j) in the input table is part of the output. We compute log-loss for each entry and the final loss is given by:
Llookup(lookup answerT , y) = N∑ i=1 min(r,c)∈Si(− log(lookup answerT [r, c]))
− 1 MC M∑ i=1 C∑ j=1 [g[i, j] == 0] log(1− lookup answerT [i, j])
where [cond] is 1 when cond is True, and 0 otherwise.
We deal with the ambiguity that occurs when the ground truth is a number and if the number also occurs in the table, by computing the final loss as the soft minimum of Lscalar and Llookup. Otherwise, the loss for an example is Lscalar when the ground truth is a number and Llookup when the ground truth matches some entries in the table. The two loss functions Lscalar and Llookup are in different scales, so we multiply Llookup by a constant factor which we set to 50.0 after a small exploration in our experiments.
Since we employ hard selection at test time, only one among scalar answer and lookup answer is modified at the last timestep. We use the variable that is set at the last timestep as the final output of the model.
3 EXPERIMENTS
We apply Neural Programmer on the WikiTableQuestions dataset (Pasupat & Liang, 2015) and compare it to different non-neural baselines including a natural language semantic parser developed by Pasupat & Liang (2015). Further, we also report results from training the sequence-tosequence model (Sutskever et al., 2014) and a modified version of the pointer networks (Vinyals et al., 2015). Our model is implemented in TensorFlow (Abadi et al., 2016) and the model takes approximately a day to train on a single Tesla K80 GPU. We use double-precision format to store the model parameters since the gradients become undefined values in single-precision format. Our code is available at https://github.com/tensorflow/models/tree/master/neural_ programmer.
3.1 DATA
We use the train, development, and test split given by Pasupat & Liang (2015). The dataset contains 11321, 2831, and 4344 examples for training, development, and testing respectively. We use their tokenization, number and date pre-processing. There are examples with answers that are neither
number answers nor phrases selected from the table. We ignore these questions during training but the model is penalized during evaluation following Pasupat & Liang (2015). The tables provided in the test set are unseen at training, hence requiring the model to adapt to unseen column names at test time. We train only on examples for which the provided table has less than 100 rows since we run out of GPU memory otherwise, but consider all examples at test time.
3.2 TRAINING DETAILS
We use T = 4 timesteps in our experiments. Words and operations are represented as 256 dimensional vectors, and the hidden vectors of the question and the history RNN are also 256 dimensional. The parameters are initialized uniformly randomly within the range [-0.1, 0.1]. We train the model using the Adam optimizer (Kingma & Ba, 2014) with mini-batches of size 20. The hyperparameter in Adam is set to 1e-6 while others are set to the default values. Since the training set is small compared to other datasets in which neural network models are usually applied, we rely on strong regularization:
• We clip the gradients to norm 1 and employ early-stopping. • The occurrences of words that appear less than 10 times in the training set are replaced by
a single unknown word token.
• We add a weight decay penalty with strength 0.0001. • We use dropout with a keep probability of 0.8 on input and output vectors of the RNN, and
selector, operation and column name representations (Srivastava et al., 2014).
• We use dropout with keep probability of 0.9 on the recurrent connections of the question RNN and history RNN using the technique from Gal & Ghahramani (2016).
• We use word-dropout (Iyyer et al., 2015) with keep probability of 0.9. Here, words in the question are randomly replaced with the unknown word token while training.
We tune the dropout rates, regularization strength, and the hyperparameter using grid search on the development data, we fix the other hyperparameters after a small exploration during initial experiments.
3.3 RESULTS
Table 1 shows the performance of our model in comparison to baselines from Pasupat & Liang (2015). The best result from Neural Programmer is achieved by an ensemble of 15 models. The only difference among these models is that the parameters of each model is initialized with a different random seed. We combine the models by averaging the predicted softmax distributions of the models at every timestep. While it is generally believed that neural network models require a large number of training examples compared to simpler linear models to get good performance, our model
achieves competitive performance on this small dataset containing only 10,000 examples with weak supervision.
We did not get better results either by using pre-trained word vectors (Mikolov et al., 2013) or by pre-training the question RNN with a language modeling objective (Dai & Le, 2015). A possible explanation is that the word vectors obtained from unsupervised learning may not be suitable to the task under consideration. For example, the learned representations of words like maximum and minimum from unsupervised learning are usually close to each other but for our task it is counterproductive. We consider replacing soft selection with hard selection and training the model with the REINFORCE algorithm (Williams, 1992). The model fails to learn in this experiment, probably because the model has to search over millions of symbolic programs for every input question making it highly unlikely to find a program that gives a reward. Hence, the parameters of the model are not updated frequently enough.
3.3.1 NEURAL NETWORK BASELINES
To understand the difficulty of the task for neural network models, we experiment with two neural network baselines: the sequence-to-sequence model (Sutskever et al., 2014) and a modified version of the pointer networks (Vinyals et al., 2015). The input to the sequence-to-sequence model is a concatenation of the table and the question, and the decoder produces the output one token at a time. We consider only examples whose input length is less than 400 to make the running time reasonable. The resulting dataset has 8, 857 and 1, 623 training and development examples respectively. The accuracy of the best model on this development set after hyperparameter tuning is only 8.9%. Next, we experiment with pointer networks to select entries in the table as the final answer. We modify pointer networks to have two-attention heads: one to select the column and the other to select entries within a column. Additionally, the model performs multiple pondering steps on the table before returning the final answer. We train this model only on lookup questions, since the model does not have a decoder to generate answers. We consider only examples whose tables have less than 100 rows resulting in training and development set consisting of 7, 534 and 1, 829 examples respectively. The accuracy of the best model on this development set after hyperparameter tuning is only 4.0%. These results confirm our intuition that discrete operations are hard to learn for neural networks particularly with small datasets in real-world settings.
3.4 ANALYSIS
3.4.1 MODEL ABLATION
Table 2 shows the impact of different model design choices on the final performance. While anonymizing phrases in the question that match some table entry seems to have a small positive effect, regularization has a much larger effect on the performance. Column selection is performed in Neelakantan et al. (2016) using only the name of a column; however, this selection procedure is insufficient in real-world settings. For example the column selected in question 3 in Table 3 does not have a corresponding phrase in the question. Hence, to select a column we additionally use a boolean feature that indicates whether an entry in that column matches some phrase in the question. Table 2 shows that the addition of this boolean feature has a significant effect on performance.
3.4.2 INDUCED PROGRAMS
Table 3 shows few examples of programs induced by Neural Programmer that yield the correct answer in the development set. The programs given in Table 3 show a few characteristics of the learned model. First, our analysis indicates that the model can adapt to unseen column names at test time. For example in Question 3, the word outcome occurs only 8 times in the training set and is replaced with the unknown word token. Second, the model does not always induce the most efficient (with respect to number of operations other than the reset operation that are picked) program to solve a task. The last 3 questions in the table can be solved using simpler programs. Finally, the model does not always induce the correct program to get the ground truth answer. For example, the last 2 programs will not result in the correct response for all input database tables. The programs would produce the correct response only when the select operation matches one entry in the table.
3.4.3 CONTRIBUTION OF DIFFERENT OPERATIONS
Table 4 shows the contribution of the different operations. The model induces a program that results in a scalar answer 30.7% of the time while the induced program is a table lookup for the remaining questions. The two most commonly used operations by the model are print and select.
3.4.4 ERROR ANALYSIS
To conclude this section, we suggest ideas to potentially improve the performance of the model. First, the oracle performance with 15 Neural Programmer models is 50.5% on the development set while averaging achieves only 37.5% implying that there is still room for improvement. Next, the accuracy of a single model on the training set is 53% which is about 20% higher than the accuracy in both the development set and the test set. This difference in performance indicates that the model suffers from significant overfitting even after employing strong regularization. It also suggests that the performance of the model could be greatly improved by obtaining more training data. Nevertheless, there are limits to the performance improvements we may reasonably expect: in particular, as shown in previous work (Pasupat & Liang, 2015), 21% of questions on a random set of 200 examples in the considered dataset are not answerable because of various issues such as annotation errors and tables requiring advanced normalization.
4 OTHER RELATED WORK
While we discuss in detail various semantic parsing and neural program induction techniques in Section 1, here we briefly describe other relevant work. Recently, Kocisky et al. (2016) develop a semi-supervised semantic parsing method that uses question-program pairs as supervision. Concurrently to our work, Liang et al. (2016) propose neural symbolic machine, a model very similar to Neural Programmer but trained using the REINFORCE algorithm (Williams, 1992). They use only 2 discrete operations and run for a total of 3 timesteps, hence inducing programs that are much simpler than ours. Neural networks have also been applied on question-answering datasets that do not require much arithmetic reasoning (Bordes et al., 2014; Iyyer et al., 2014; Sukhbaatar et al., 2015; Peng et al., 2015; Hermann et al., 2015; Kumar et al., 2016). Wang & Jiang (2016) use a neural network model to get state-of-the-art results on a reading comprehension task (Rajpurkar et al., 2016).
5 CONCLUSION
In this paper, we enhance Neural Programmer to work with weaker supervision signals to make it more broadly applicable. Soft selection during training enables the model to actively explore the space of programs by backpropagation with superior sample complexity. In our experiments, we show that the model achieves performance comparable to a state-of-the-art traditional semantic parser even though the training set contains only 10,000 examples. To our knowledge, this is the first instance of a weakly supervised, end-to-end neural network model that induces programs on a real-world dataset.
Acknowledgements We are grateful to Panupong Pasupat for answering numerous questions about the dataset, and providing pre-processed version of the dataset and the output of the semantic parser. We thank David Belanger, Samy Bengio, Greg Corrado, Andrew Dai, Jeff Dean, Nando de Freitas, Shixiang Gu, Navdeep Jaitly, Rafal Jozefowicz, Ashish Vaswani, Luke Vilnis, Yuan Yu and Barret Zoph for their suggestions and the Google Brain team for the support. Arvind Neelakantan is supported by a Google PhD fellowship in machine learning.

The paper presents an end-to-end neural network model for the problem of designing natural language interfaces for database queries. The proposed approach uses only weak supervision signals to learn the parameters of the model. Unlike in traditional approaches, where the problem is solved by semantically parsing a natural language query into logical forms and executing those logical forms over the given data base, the proposed approach trains a neural network in an end-to-end manner which goes directly from the natural language query to the final answer obtained by processing the data base. This is achieved by formulating a collection of operations to be performed over the data base as continuous operations, the distributions over which is learnt using the now-standard soft attention mechanisms. The model is validated on the smallish WikiTableQuestions dataset, where the authors show that a single model performs worse than the approach which uses the traditional Semantic Parsing technique. However an ensemble of 15 models (trained in a variety of ways) results in comparable performance to the state of the art.
I feel that the paper proposes an interesting solution to the hard problem of learning natural language interfaces for data bases. The model is an extension of the previously proposed models of Neelakantan 2016. The experimental section is rather weak though. The authors only show their model work on a single smallish dataset. Would love to see more ablation studies of their model and comparison against fancier version of memnns (i do not buy their initial response to not testing against memory networks).
I do have a few objections though.
-- The details of the model are rather convoluted and the Section 2.1 is not very clearly written. In particular with the absence of the accompanying code the model will be super hard to replicate. I wish the authors do a better job in explaining the details as to how exactly the discrete operations are modeled, what is the role of the "row selector", the "scalar answer" and the "lookup answer" etc.
-- The authors do a full attention over the entire database. Do they think this approach would scale when the data bases are huge (millions of rows)? Wish they experimented with larger datasets as well.