Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also learns separate policies for maximising many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880% expert human performance, and a challenging suite of first-person, three-dimensional Labyrinth tasks leading to a mean speedup in learning of 10× and averaging 87% expert human performance on Labyrinth. Natural and artificial agents live in a stream of sensorimotor data. At each time step t, the agent receives observations ot and executes actions at. These actions influence the future course of the sensorimotor stream. In this paper we develop agents that learn to predict and control this stream, by solving a host of reinforcement learning problems, each focusing on a distinct feature of the sensorimotor stream. Our hypothesis is that an agent that can flexibly control its future experiences will also be able to achieve any goal with which it is presented, such as maximising its future rewards. The classic reinforcement learning paradigm focuses on the maximisation of extrinsic reward. However, in many interesting domains, extrinsic rewards are only rarely observed. This raises questions of what and how to learn in their absence. Even if extrinsic rewards are frequent, the sensorimotor stream contains an abundance of other possible learning targets. Traditionally, unsupervised learning attempts to reconstruct these targets, such as the pixels in the current or subsequent frame. It is typically used to accelerate the acquisition of a useful representation. In contrast, our learning objective is to predict and control features of the sensorimotor stream, by treating them as pseudorewards for reinforcement learning. Intuitively, this set of tasks is more closely matched with the agent’s long-term goals, potentially leading to more useful representations. Consider a baby that learns to maximise the cumulative amount of red that it observes. To correctly predict the optimal value, the baby must understand how to increase “redness” by various means, including manipulation (bringing a red object closer to the eyes); locomotion (moving in front of a red object); and communication (crying until the parents bring a red object). These behaviours are likely to recur for many other goals that the baby may subsequently encounter. No understanding of these behaviours is required to simply reconstruct the redness of current or subsequent images. Our architecture uses reinforcement learning to approximate both the optimal policy and optimal value function for many different pseudo-rewards. It also makes other auxiliary predictions that serve to focus the agent on important aspects of the task. These include the long-term goal of predicting cumulative extrinsic reward as well as short-term predictions of extrinsic reward. To learn more efficiently, our agents use an experience replay mechanism to provide additional updates ∗Joint first authors. Ordered alphabetically by first name.
Natural and artificial agents live in a stream of sensorimotor data. At each time step t, the agent receives observations ot and executes actions at. These actions influence the future course of the sensorimotor stream. In this paper we develop agents that learn to predict and control this stream, by solving a host of reinforcement learning problems, each focusing on a distinct feature of the sensorimotor stream. Our hypothesis is that an agent that can flexibly control its future experiences will also be able to achieve any goal with which it is presented, such as maximising its future rewards.
The classic reinforcement learning paradigm focuses on the maximisation of extrinsic reward. However, in many interesting domains, extrinsic rewards are only rarely observed. This raises questions of what and how to learn in their absence. Even if extrinsic rewards are frequent, the sensorimotor stream contains an abundance of other possible learning targets. Traditionally, unsupervised learning attempts to reconstruct these targets, such as the pixels in the current or subsequent frame. It is typically used to accelerate the acquisition of a useful representation. In contrast, our learning objective is to predict and control features of the sensorimotor stream, by treating them as pseudorewards for reinforcement learning. Intuitively, this set of tasks is more closely matched with the agent’s long-term goals, potentially leading to more useful representations.
Consider a baby that learns to maximise the cumulative amount of red that it observes. To correctly predict the optimal value, the baby must understand how to increase “redness” by various means, including manipulation (bringing a red object closer to the eyes); locomotion (moving in front of a red object); and communication (crying until the parents bring a red object). These behaviours are likely to recur for many other goals that the baby may subsequently encounter. No understanding of these behaviours is required to simply reconstruct the redness of current or subsequent images.
Our architecture uses reinforcement learning to approximate both the optimal policy and optimal value function for many different pseudo-rewards. It also makes other auxiliary predictions that serve to focus the agent on important aspects of the task. These include the long-term goal of predicting cumulative extrinsic reward as well as short-term predictions of extrinsic reward. To learn more efficiently, our agents use an experience replay mechanism to provide additional updates
∗Joint first authors. Ordered alphabetically by first name.
⇡V ⇡V⇡V⇡V
0 0 0 +1
(c) Reward Prediction
rt } Replay Buffer
VVV
+RRR
(d) Value Function Replayt⌧ t⌧+1 t⌧+2 t⌧+3
t⌧ 3 t⌧ 2 t⌧ 1
r⌧ Skewed sampling
(b) Pixel Control
Qaux
V
…
Environment
… …
ot
Agent LSTM
Agent ConvNet
Aux DeConvNet
Aux FC net
(a) Base A3C Agent
Figure 1: Overview of the UNREAL agent. (a) The base agent is a CNN-LSTM agent trained on-policy with the A3C loss (Mnih et al., 2016). Observations, rewards, and actions are stored in a small replay buffer which encapsulates a short history of agent experience. This experience is used by auxiliary learning tasks. (b) Pixel Control – auxiliary policies Qaux are trained to maximise change in pixel intensity of different regions of the input. The agent CNN and LSTM are used for this task along with an auxiliary deconvolution network. This auxiliary control task requires the agent to learn how to control the environment. (c) Reward Prediction – given three recent frames, the network must predict the reward that will be obtained in the next unobserved timestep. This task network uses instances of the agent CNN, and is trained on reward biased sequences to remove the perceptual sparsity of rewards. (d) Value Function Replay – further training of the value function using the agent network is performed to promote faster value iteration. Further visualisation of the agent can be found in https://youtu.be/Uz-zGYrYEjA
to the critics. Just as animals dream about positively or negatively rewarding events more frequently (Olafsdottir et al., 2015; Schacter et al., 2012), our agents preferentially replay sequences containing rewarding events.
Importantly, both the auxiliary control and auxiliary prediction tasks share the convolutional neural network and LSTM that the base agent uses to act. By using this jointly learned representation, the base agent learns to optimise extrinsic reward much faster and, in many cases, achieves better policies at the end of training.
This paper brings together the state-of-the-art Asynchronous Advantage Actor-Critic (A3C) framework (Mnih et al., 2016), outlined in Section 2, with auxiliary control tasks and auxiliary reward tasks, defined in sections Section 3.1 and Section 3.2 respectively. These auxiliary tasks do not require any extra supervision or signals from the environment than the vanilla A3C agent. The result is our UNsupervised REinforcement and Auxiliary Learning (UNREAL) agent (Section 3.4)
In Section 4 we apply our UNREAL agent to a challenging set of 3D-vision based domains known as the Labyrinth (Mnih et al., 2016), learning solely from the raw RGB pixels of a first-person view. Our agent significantly outperforms the baseline agent using vanilla A3C, even when the baseline was augmented with an unsupervised reconstruction loss, in terms of speed of learning, robustness to hyperparameters, and final performance. The result is an agent which on average achieves 87% of expert human-normalised score, compared to 54% with A3C, and on average 10× faster than A3C. Our UNREAL agent also significantly outperforms the previous state-of-the-art in the Atari domain.
1 RELATED WORK
A variety of reinforcement learning architectures have focused on learning temporal abstractions, such as options (Sutton et al., 1999b), with policies that may maximise pseudo-rewards (Konidaris & Barreto, 2009; Silver & Ciosek, 2012). The emphasis here has typically been on the development of temporal abstractions that facilitate high-level learning and planning. In contrast, our agents do
not make any direct use of the pseudo-reward maximising policies that they learn (although this is an interesting direction for future research). Instead, they are used solely as auxiliary objectives for developing a more effective representation.
The Horde architecture (Sutton et al., 2011) also applied reinforcement learning to identify value functions for a multitude of distinct pseudo-rewards. However, this architecture was not used for representation learning; instead each value function was trained separately using distinct weights.
The UVFA architecture (Schaul et al., 2015a) is a factored representation of a continuous set of optimal value functions, combining features of the state with an embedding of the pseudo-reward function. Initial work on UVFAs focused primarily on architectural choices and learning rules for these continuous embeddings. A pre-trained UVFA representation was successfully transferred to novel pseudo-rewards in a simple task.
Similarly, the successor representation (Dayan, 1993; Barreto et al., 2016; Kulkarni et al., 2016) factors a continuous set of expected value functions for a fixed policy, by combining an expectation over features of the state with an embedding of the pseudo-reward function. Successor representations have been used to transfer representations from one pseudo-reward to another (Barreto et al., 2016) or to different scales of reward (Kulkarni et al., 2016).
Another, related line of work involves learning models of the environment (Schmidhuber, 2010; Xie et al., 2015; Oh et al., 2015). Although learning environment models as auxiliary tasks could improve RL agents (e.g. Lin & Mitchell (1992); Li et al. (2015)), this has not yet been shown to work in rich visual environments.
More recently, auxiliary predictions tasks have been studied in 3D reinforcement learning environments. Lample & Chaplot (2016) showed that predicting internal features of the emulator, such as the presence of an enemy on the screen, is beneficial. Mirowski et al. (2016) study auxiliary prediction of depth in the context of navigation.
2 BACKGROUND
We assume the standard reinforcement learning setting where an agent interacts with an environment over a number of discrete time steps. At time t the agent receives an observation ot along with a reward rt and produces an action at. The agent’s state st is a function of its experience up until time t, st = f(o1, r1, a1, ..., ot, rt). The n-step return Rt:t+n at time t is defined as the discounted sum of rewards, Rt:t+n = ∑n i=1 γ
i−1rt+i. The value function is the expected return from state s, V π(s) = E [Rt:∞|st = s, π], when actions are selected accorded to a policy π(a|s). The actionvalue functionQπ(s, a) = E [Rt:∞|st = s, at = a, π] is the expected return following action a from state s.
Value-based reinforcement learning algorithms, such as Q-learning (Watkins, 1989), or its deep learning instantiations DQN (Mnih et al., 2015) and asynchronous Q-learning (Mnih et al., 2016), approximate the action-value function Q(s, a; θ) using parameters θ, and then update parameters to minimise the mean-squared error, for example by optimising an n-step lookahead loss (Peng & Williams, 1996), LQ = E [ (Rt:t+n + γ n maxa′ Q(s ′, a′; θ−)−Q(s, a; θ))2 ] ; where θ− are
previous parameters and the optimisation is with respect to θ.
Policy gradient algorithms adjust the policy to maximise the expected reward, Es∼π [R1:∞], using the gradient ∂Es∼π [R1:∞]∂θ = E [ ∂ ∂θ log π(a|s)(Qπ(s, a)− V π(s)) ] (Watkins, 1989; Sutton et al., 1999a); in practice the true value functions Qπ and V π are substituted with approximations. The Asynchronous Advantage Actor-Critic (A3C) algorithm (Mnih et al., 2016) constructs an approximation to both the policy π(a|s, θ) and the value function V (s, θ) using parameters θ. Both policy and value are adjusted towards an n-step lookahead value, Rt:t+n + γnV (st+n+1, θ), using an entropy regularisation penalty, LA3C ≈ LVR + Lπ − Es∼π [αH(π(s, ·, θ)], where LVR = Es∼π [ (Rt:t+n + γ nV (st+n+1, θ −)− V (st, θ))2 ] .
In A3C many instances of the agent interact in parallel with many instances of the environment, which both accelerates and stabilises learning. The A3C agent architecture we build on uses an LSTM to jointly approximate both policy π and value function V , given the entire history of experience as inputs (see Figure 1 (a)).
3 AUXILIARY TASKS FOR REINFORCEMENT LEARNING
In this section we incorporate auxiliary tasks into the reinforcement learning framework in order to promote faster training, more robust learning, and ultimately higher performance for our agents. Section 3.1 introduces the use of auxiliary control tasks, Section 3.2 describes the addition of reward focussed auxiliary tasks, and Section 3.4 describes the complete UNREAL agent combining these auxiliary tasks.
3.1 AUXILIARY CONTROL TASKS
The auxiliary control tasks we consider are defined as additional pseudo-reward functions in the environment the agent is interacting with. We formally define an auxiliary control task c by a reward function r(c) : S × A → R, where S is the space of possible states and A is the space of available actions. The underlying state space S includes both the history of observations and rewards as well as the state of the agent itself, i.e. the activations of the hidden units of the network.
Given a set of auxiliary control tasks C, let π(c) be the agent’s policy for each auxiliary task c ∈ C and let π be the agent’s policy on the base task. The overall objective is to maximise total performance across all these auxiliary tasks,
argmax θ Eπ[R1:∞] + λC ∑ c∈C Eπc [R (c) 1:∞], (1)
where, R(c)t:t+n = ∑n k=1 γ k−1r(c)t+k is the discounted return for auxiliary reward r (c), and θ is the set of parameters of π and all π(c)’s. By sharing some of the parameters of π and all π(c) the agent must balance improving its performance with respect to the global reward rt with improving performance on the auxiliary tasks.
In principle, any reinforcement learning method could be applied to maximise these objectives. However, to efficiently learn to maximise many different pseudo-rewards simultaneously in parallel from a single stream of experience, it is necessary to use off-policy reinforcement learning. We focus on value-based RL methods that approximate the optimal action-values by Qlearning. Specifically, for each control task c we optimise an n-step Q-learning loss L(c)Q = E [( Rt:t+n + γ n maxa′ Q (c)(s′, a′, θ−)−Q(c)(s, a, θ) )2] , as described in Mnih et al. (2016).
While many types of auxiliary reward functions can be defined from these quantities we focus on two specific types:
• Pixel changes - Changes in the perceptual stream often correspond to important events in an environment. We train agents that learn a separate policy for maximally changing the pixels in each cell of an n× n non-overlapping grid placed over the input image. We refer to these auxiliary tasks as pixel control. See Section 4 for a complete description. • Network features - Since the policy or value networks of an agent learn to extract task-
relevant high-level features of the environment (Mnih et al., 2015; Zahavy et al., 2016; Silver et al., 2016) they can be useful quantities for the agent to learn to control. Hence, the activation of any hidden unit of the agent’s neural network can itself be an auxiliary reward. We train agents that learn a separate policy for maximally activating each of the units in a specific hidden layer. We refer to these tasks as feature control.
The Figure 1 (b) shows an A3C agent architecture augmented with a set of auxiliary pixel control tasks. In this case, the base policy π shares both the convolutional visual stream and the LSTM with the auxiliary policies. The output of the auxiliary network head is an Nact × n × n tensor Qaux where Qaux(a, i, j) represents the network’s current estimate of the optimal discounted expected change in cell (i, j) of the input after taking action a. We exploit the spatial nature of the auxiliary tasks by using a deconvolutional neural network to produce the auxiliary values Qaux.
3.2 AUXILIARY REWARD TASKS
In addition to learning generally about the dynamics of the environment, an agent must learn to maximise the global reward stream. To learn a policy to maximise rewards, an agent requires features
that recognise states that lead to high reward and value. An agent with a good representation of rewarding states, will allow the learning of good value functions, and in turn should allow the easy learning of a policy.
However, in many interesting environments reward is encountered very sparsely, meaning that it can take a long time to train feature extractors adept at recognising states which signify the onset of reward. We want to remove the perceptual sparsity of rewards and rewarding states to aid the training of an agent, but to do so in a way which does not introduce bias to the agent’s policy.
To do this, we introduce the auxiliary task of reward prediction – that of predicting the onset of immediate reward given some historical context. This task consists of processing a sequence of consecutive observations, and requiring the agent to predict the reward picked up in the subsequent unseen frame. This is similar to value learning focused on immediate reward (γ = 0).
Unlike learning a value function, which is used to estimate returns and as a baseline while learning a policy, the reward predictor is not used for anything other than shaping the features of the agent. This keeps us free to bias the data distribution, therefore biasing the reward predictor and feature shaping, without biasing the value function or policy.
We train the reward prediction task on sequences Sτ = (sτ−k, sτ−k+1, . . . , sτ−1) to predict the reward rτ , and sample Sτ from the experience of our policy π in a skewed manner so as to overrepresent rewarding events (presuming rewards are sparse within the environment). Specifically, we sample such that zero rewards and non-zero rewards are equally represented, i.e. the predicted probability of a non-zero reward is P (rτ 6= 0) = 0.5. The reward prediction is trained to minimise a loss LRP. In our experiments we use a multiclass cross-entropy classification loss across three classes (zero, positive, or negative reward), although a mean-squared error loss is also feasible.
The auxiliary reward predictions may use a different architecture to the agent’s main policy. Rather than simply “hanging” the auxiliary predictions off the LSTM, we use a simpler feedforward network that concatenates a stack of states Sτ after being encoded by the agent’s CNN, see Figure 1 (c). The idea is to simplify the temporal aspects of the prediction task in both the future direction (focusing only on immediate reward prediction rather than long-term returns) and past direction (focusing only on immediate predecessor states rather than the complete history); the features discovered in this manner are shared with the primary LSTM (via shared weights in the convolutional encoder) to enable the policy to be learned more efficiently.
3.3 EXPERIENCE REPLAY
Experience replay has proven to be an effective mechanism for improving both the data efficiency and stability of deep reinforcement learning algorithms (Mnih et al., 2015). The main idea is to store transitions in a replay buffer, and then apply learning updates to sampled transitions from this buffer.
Experience replay provides a natural mechanism for skewing the distribution of reward prediction samples towards rewarding events: we simply split the replay buffer into rewarding and nonrewarding subsets, and replay equally from both subsets. The skewed sampling of transitions from
a replay buffer means that rare rewarding states will be oversampled, and learnt from far more frequently than if we sampled sequences directly from the behaviour policy. This approach can be viewed as a simple form of prioritised replay (Schaul et al., 2015b).
In addition to reward prediction, we also use the replay buffer to perform value function replay (see Figure 1). This amounts to resampling recent historical sequences from the behaviour policy distribution and performing extra value function regression in addition to the on-policy value function regression in A3C. By resampling previous experience, and randomly varying the temporal position of the truncation window over which the n-step return is computed, value function replay performs value iteration and exploits newly discovered features shaped by reward prediction. We do not skew the distribution for this case.
Experience replay is also used to increase the efficiency and stability of the auxiliary control tasks. Q-learning updates are applied to sampled experiences that are drawn from the replay buffer, allowing features to be developed extremely efficiently.
3.4 UNREAL AGENT
The UNREAL algorithm combines the benefits of two separate, state-of-the-art approaches to deep reinforcement learning. The primary policy is trained with A3C (Mnih et al., 2016): it learns from parallel streams of experience to gain efficiency and stability; it is updated online using policy gradient methods; and it uses a recurrent neural network to encode the complete history of experience. This allows the agent to learn effectively in partially observed environments.
The auxiliary tasks are trained on very recent sequences of experience that are stored and randomly sampled; these sequences may be prioritised (in our case according to immediate rewards) (Schaul et al., 2015b); these targets are trained off-policy by Q-learning; and they may use simpler feedforward architectures. This allows the representation to be trained with maximum efficiency.
The UNREAL algorithm optimises a single combined loss function with respect to the joint parameters of the agent, θ, that combines the A3C loss LA3C together with an auxiliary control loss LPC = ∑ c L (c) Q , auxiliary reward prediction loss LRP and replayed value loss LVR,
LUNREAL(θ) = LA3C + λVRLVR + λPC ∑ c L(c)Q + λRPLRP (2)
where λVR, λPC, λRP are weighting terms on the individual loss components.
In practice, the loss is broken down into separate components that are computed either on-policy, directly from experience; or off-policy, on replayed transitions. Specifically, the A3C loss LA3C is minimised on-policy; while the value function loss LVR is optimised from replayed data, in addition to the A3C loss (of which it is one component, see Section 2). The auxiliary control loss LPC is optimised off-policy from replayed data, by n-step Q-learning. Finally, the reward loss LRP is optimised from rebalanced replay data.
4 EXPERIMENTS
In this section we give the results of experiments performed on the 3D environment Labyrinth in Section 4.1 and Atari in Section 4.2.
In all our experiments we used an A3C CNN-LSTM agent as our baseline and the UNREAL agent along with its ablated variants added auxiliary outputs and losses to this base agent. The agent is trained on-policy with 20-step returns and the auxiliary tasks are performed every 20 environment steps, corresponding to every update of the base A3C agent. The replay buffer stores the most recent 2k observations, actions, and rewards taken by the base agent. In Labyrinth we use the same set of 17 discrete actions for all games and on Atari the action set is game dependent (between 3 and 18 discrete actions). The full implementation details can be found in Section B.
4.1 LABYRINTH RESULTS
Labyrinth (see Figure 2) is a first-person 3D game platform extended from OpenArena (contributors, 2005), which is itself based on Quake3 (id software, 1999). Labyrinth is comparable to other first-
person 3D game platforms for AI research like VizDoom (Kempka et al., 2016) or Minecraft (Tessler et al., 2016). However, in comparison, Labyrinth has considerably richer visuals and more realistic physics. Textures in Labyrinth are often dynamic (animated) so as to convey a game world where walls and floors shimmer and pulse, adding significant complexity to the perceptual task. The action space allows for fine-grained pointing in a fully 3D world. Labyrinth also supports continuous motion unlike the Minecraft platform of (Oh et al., 2016), which is a 3D grid world.
We evaluated agent performance on 13 Labyrinth levels that tested a range of different agent abilities. A top-down visualization showing the layout of each level can be found in Figure 9 of the Appendix. A gallery of example images from the first-person perspective of the agent are in Figure 10 of the Appendix. The levels can be divided into four categories:
1. Simple fruit gathering levels with a static map (seekavoid arena 01 and stairway to melon 01). The goal of these levels is to collect apples (small positive reward) and melons (large positive reward) while avoiding lemons (small negative reward).
2. Navigation levels with a static map layout (nav maze static 0{1, 2, 3} and nav maze random goal 0{1, 2, 3}). These levels test the agent’s ability to find their way to a goal in a fixed maze that remains the same across episodes. The starting location is random. In this case, agents could encode the structure of the maze in network weights. In the random goal variant, the location of the goal changes in every episode. The optimal policy is to find the goal’s location at the start of each episode and then use long-term knowledge of the maze layout to return to it as quickly as possible from any location. The static variant is simpler in that the goal location is always fixed for all episodes and only the agent’s starting location changes so the optimal policy does not require the first step of exploring to find the current goal location.
3. Procedurally-generated navigation levels requiring effective exploration of a new maze generated on-the-fly at the start of each episode (nav maze all random 0{1, 2, 3}). These levels test the agent’s ability to effectively explore a totally new environment. The optimal
policy would begin by exploring the maze to rapidly learn its layout and then exploit that knowledge to repeatedly return to the goal as many times as possible before the end of the episode (between 60 and 300 seconds).
4. Laser-tag levels requiring agents to wield laser-like science fiction gadgets to tag bots controlled by the game’s in-built AI (lt horse shoe color and lt hallway slope). A reward of 1 is delivered whenever the agent tags a bot by reducing its shield to 0. These levels approximate the default OpenArena/Quake3 gameplay mode. In lt hallway slope there is a sloped arena, requiring the agent to look up and down. In lt horse shoe color, the colors and textures of the bots are randomly generated at the start of each episode. This prevents agents from relying on color for bot detection. These levels test aspects of fine-control (for aiming), planning (to anticipate where bots are likely to move), strategy (to control key areas of the map such as gadget spawn points), and robustness to the substantial visual complexity arising from the large numbers of independently moving objects (gadget projectiles and bots).
4.1.1 RESULTS
We compared the full UNREAL agent to a basic A3C LSTM agent along with several ablated versions of UNREAL with different components turned off. A video of the final agent performance, as well as visualisations of the activations and auxiliary task outputs can be viewed at https://youtu.be/Uz-zGYrYEjA.
Figure 3 (top left) shows curves of mean human-normalised scores over the 13 Labyrinth levels. Adding each of our proposed auxiliary tasks to an A3C agent substantially improves the performance. Combining different auxiliary tasks leads to further improvements over the individual auxiliary tasks. The UNREAL agent, which combines all three auxiliary tasks, achieves more than twice the final human-normalised mean performance of A3C, increasing from 54% to 87% (45% to 92% for median performance). This includes a human-normalised score of 116% on lt hallway slope and 100% on nav maze random goal 02.
Perhaps of equal importance, aside from final performance on the games, UNREAL is significantly faster at learning and therefore more data efficient, achieving a mean speedup of the number of steps to reach A3C best performance of 10× (median 11×) across all levels and up to 18× on nav maze random goal 02. This translates in a drastic improvement in the data efficiency of UNREAL over A3C, requiring less than 10% of the data to reach the final performance of A3C. We can also measure the robustness of our learning algorithms to hyperparameters by measuring the performance over all hyperparameters (namely learning rate and entropy cost). This is shown in Figure 3 Top Right: every auxiliary task in our agent improves robustness. A breakdown of the performance of A3C, UNREAL and UNREAL without pixel control on the individual Labyrinth levels is shown in Figure 4.
Unsupervised Reinforcement Learning In order to better understand the benefits of auxiliary control tasks we compared it to two simple baselines on three Labyrinth levels. The first baseline was A3C augmented with a pixel reconstruction loss, which has been shown to improve performance on 3D environments (Kulkarni et al., 2016). The second baseline was A3C augmented with an input change prediction loss, which can be seen as simply predicting the immediate auxiliary reward instead of learning to control. Finally, we include preliminary results for A3C augmented with the feature control auxiliary task on one of the levels. We retuned the hyperparameters of all methods (including learning rate and the weight placed on the auxiliary loss) for each of the three Labyrinth levels. Figure 5 shows the learning curves for the top 5 hyperparameter settings on three Labyrinth navigation levels. The results show that learning to control pixel changes is indeed better than simply predicting immediate pixel changes, which in turn is better than simply learning to reconstruct the input. In fact, learning to reconstruct only led to faster initial learning and actually made the final scores worse when compared to vanilla A3C. Our hypothesis is that input reconstruction hurts final performance because it puts too much focus on reconstructing irrelevant parts of the visual input instead of visual cues for rewards, which rewarding objects are rarely visible. We saw a substantial improvement from including the feature control auxiliary task, which was only slightly worse than for pixel control. Combining feature control with other auxiliary tasks is a promising future direction.
4.2 ATARI
We applied the UNREAL agent as well as UNREAL without pixel control to 57 Atari games from the Arcade Learning Environment (Bellemare et al., 2012) domain. We use the same evaluation protocol as for our Labyrinth experiments where we evaluate 50 different random hyper parameter settings (learning rate and entropy cost) on each game. The results are shown in the bottom row of Figure 3. The left side shows the average performance curves of the top 3 agents for all three methods the right half shows sorted average human-normalised scores for each hyperparameter setting. More detailed learning curves for individual levels can be found in Figure 6. We see that UNREAL surpasses the current state-of-the-art agents, i.e. A3C and Prioritized Dueling DQN (Wang et al., 2016), across all levels attaining 880% mean and 250% median performance. Notably, UNREAL is also substantially more robust to hyper parameter settings than A3C.
5 CONCLUSION
We have shown how augmenting a deep reinforcement learning agent with auxiliary control and reward prediction tasks can drastically improve both data efficiency and robustness to hyperparameter settings. Most notably, our proposed UNREAL architecture more than doubled the previous stateof-the-art results on the challenging set of 3D Labyrinth levels, bringing the average scores to over 87% of human scores. The same UNREAL architecture also significantly improved both the learning speed and the robustness of A3C over 57 Atari games.
ACKNOWLEDGEMENTS
We thank Charles Beattie, Julian Schrittwieser, Marcus Wainwright, and Stig Petersen for environment design and development, and Amir Sadik and Sarah York for expert human game testing. We also thank Joseph Modayil, Andrea Banino, Hubert Soyer, Razvan Pascanu, and Raia Hadsell for many helpful discussions.
A ATARI GAMES
B IMPLEMENTATION DETAILS
The input to the agent at each timestep was an 84× 84 RGB image. All agents processed the input with the convolutional neural network (CNN) originally used for Atari by Mnih et al. (2013). The network consists of two convolutional layers. The first one has 16 8× 8 filters applied with stride 4, while the second one has 32 4 × 4 filters with stride 2. This is followed by a fully connected layer with 256 units. All three layers are followed by a ReLU non-linearity. All agents used an LSTM with forget gates (Gers et al., 2000) with 256 cells which take in the CNN-encoded observation concatenated with the previous action taken and current reward. The policy and value function are linear projections of the LSTM output. The agent is trained with 20-step unrolls. The action space of the agent in the environment is game dependent for Atari (between 3 and 18 discrete actions), and 17 discrete actions for Labyrinth. Labyrinth runs at 60 frames-per-second. We use an action repeat of four, meaning that each action is repeated four times, with the agent receiving the final fourth frame as input to the next processing step.
For the pixel control auxiliary tasks we trained policies to control the central 80 × 80 crop of the inputs. The cropped region was subdivided into a 20× 20 grid of non-overlapping 4× 4 cells. The instantaneous reward in each cell was defined as the average absolute difference from the previous frame, where the average is taken over both pixels and channels in the cell. The output tensor of auxiliary values, Qaux, is produced from the LSTM outputs by a deconvolutional network. The LSTM outputs are first mapped to a 32 × 7 × 7 spatial feature map with a linear layer followed by a ReLU. This is followed by a doconvolutional layer of 32 3 × 3 filters and a ReLU, resulting in a 32 × 9 × 9 feature map. Deconvolution layers with 1 and Nact filters of size 4 × 4 and stride 2 map the 32 × 9 × 9 into a value tensor and an advantage tensor respectively. The spatial map is then decoded into Q-values using the dueling parametrization (Wang et al., 2016) producing the Nact × 20× 20 output Qaux. There is a final ReLU nonlinearity on the Qaux output. The architecture for feature control was similar. We learned to control the second hidden layer, which is a spatial feature map with size 32× 9× 9. Similarly to pixel control, we exploit the spatial structure in the data and used a deconvolutional network to produce Qaux from the LSTM outputs.
The reward prediction task is performed on a sequence of three observations, which are fed through three instances of the agent’s CNN. The three encoded CNN outputs are concatenated and fed through a fully connected layer of 128 units with ReLU activations, followed by a final linear threeclass classifier and softmax. The reward is predicted as one of three classes: positive, negative, or zero and trained with a task weight λRP = 1. The value function replay is performed on a sequence of length 20 with a task weight λVR = 1.
The auxiliary tasks are performed every 20 environment steps, corresponding to every update of the base A3C agent, once the replay buffer has filled with agent experience. The replay buffer stores the most recent 2k observations, actions, and rewards taken by the base agent.
The agents are optimised over 32 asynchronous threads with shared RMSprop (Mnih et al., 2016). The learning rates are sampled from a log-uniform distribution between 0.0001 and 0.005. The entropy costs are sampled from the log-uniform distribution between 0.0005 and 0.01. Task weight λPC is sampled from log-uniform distribution between 0.01 and 0.1 for Labyrinth and 0.0001 and 0.01 for Atari (since Atari games are not homogeneous in terms of pixel intensities changes, thus we need to fit this normalization factor).
C RANDOMNESS ROBUSTNESS
Each agent was trained with 45 randomly sampled values of hyperparameters. Each of them also starts with a different random seed (however, due to asynchronous nature of A3C this does not determinise the learning procedure).
Previous sections showed that the UNREAL agent is more robust to the choice of hyperparameters than A3C. To present an even clearer picture of this effect, we show learning curves averaged over all hyperparamters/seeds used in the experiments in Figure 7. It is worth noting, that standard error for such curves is not increased despite adding our auxiliary tasks.
We also include scatter plots of averaged final human normalised performance with respect to the two main hyperparameters (learning rate and entropy cost) in Figure 8. The final performance across all levels varies rather smoothly across similar hyperparameters, showing that learning is not significantly affected by random seeds. The only significant inconsistency, which can be spotted around (−3,−3) point in UNREAL plot is an effect of the third hyperparamer - λPC, which differs a lot between these runs.
D RAW ATARI SCORES
E LABYRINTH LEVELS

This paper is about improving feature learning in deep reinforcement learning, by augmenting the main policy's optimization problem with terms corresponding to (domain-independent) auxiliary tasks. These tasks are about control (learning other policies that attempt to maximally modify the state space, i.e. here the pixels), immediate reward prediction, and value function replay. Except for the latter, these auxiliary tasks are only used to help shape the features (by sharing the CNN+LSTM feature extraction network). Experiments show the benefits of this approach on Atari and Labyrinth problems, with in particular much better data efficiency than A3C.
The paper is well written, ideas are sound, and results pretty convincing, so to me this is a clear acceptance. At high level I only have few things to say, none being of major concern:
- I believe you should say something about the extra computational cost of optimizing these auxiliary tasks. How much do you lose in terms of training speed? Which are the most costly components?
- If possible, please try to make it clearer in the abstract / intro that the agent is learning different policies for each task. When I read in the abstract that the agent "also maximises many other pseudo-reward functions simultaneously by reinforcement learning", my first understanding was that it learned a single policy to optimize all rewards together, and I realized my mistake only when reaching eq. 1.
- The "feature control" idea is not validated empirically (the preliminary experiment in Fig. 5 is far from convincing as it only seems to help slightly initially). I like that idea but I am worried by the fact the task is changing during learning, since the extracted features are being modified. There might be stability / convergence issues at play here.
- Since as you mentioned, "the performance of our agents is still steadily improving", why not keep them going to see how far they go? (at least the best ones)
- Why aren't the auxiliary tasks weight parameters (the lambda_*) hyperparameters to optimize? Were there any experiments to validate that using 1 was a good choice?
- Please mention the fact that auxiliary tasks are not trained with "true" Q-Learning since they are trained off-policy with more than one step of empirical rewards (as discussed in the OpenReview comments)
Minor stuff:
- "Policy gradient algorithms adjust the policy to maximise the expected reward, L_pi = -..." => that's actually a loss to be minimized
- In eq. 1 lambda_c should be within the sum
- Just below eq. 1 r_t^(c) should be r_t+k^(c)
- Figure 2 does not seem to be referenced in the text, also Figure 1(d) should be referenced in 3.3
- "the features discovered in this manner is shared" => are shared
- The text around eq. 2 refers to the loss L_PC but that term is not defined and is not (explicitly) in eq. 2
- Please explain what "Clip" means for dueling networks in the legend of Figure 3
- I would have liked to see more ablated versions on Atari, to see in particular if the same patterns of individual contribution as on Labyrinth were observed
- In the legend of Figure 3 the % mentioned are for Labyrinth, which is not clear from the text.
- In 4.1.2: "Figure 3 (right) shows..." => it is actually the top left plot of the figure. Also later "This is shown in Figure 3 Top" should be Figure 3 Top Right.
- "Figure 5 shows the learning curves for the top 5 hyperparameter settings on three Labyrinth navigation levels" => I think it is referring to the left and middle plots of the figure, so only on two levels (the text above might also need fixing)
- In 4.2: "The left side shows the average performance curves of the top 5 agents for all three methods the right half shows..." => missing a comma or something after "methods"
- Appendix: "Further details are included in the supplementary materials." => where are they?
- What is the value of lambda_PC? (=1 I guess?)
[Edit] I know some of my questions were already answered in Comments, no need to re-answer them


Recent research in the cognitive process of reading hypothesized that we do not read words by sequentially recognizing letters, but rather by identifing openbigrams, i.e. couple of letters that are not necessarily next to each other. In this paper, we evaluate an handwritten word recognition method based on original open-bigrams representation. We trained Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) to predict open-bigrams rather than characters, and we show that such models are able to learn the long-range, complicated and intertwined dependencies in the input signal, necessary to the prediction. For decoding, we decomposed each word of a large vocabulary into the set of constituent bigrams, and apply a simple cosine similarity measure between this representation and the bagged RNN prediction to retrieve the vocabulary word. We compare this method to standard word recognition techniques based on sequential character recognition. Experiments are carried out on two public databases of handwritten words (Rimes and IAM), an the results with our bigram decoder are comparable to more conventional decoding methods based on sequences of letters.
Recent research in the cognitive process of reading hypothesized that we do not read words by sequentially recognizing letters, but rather by identifing openbigrams, i.e. couple of letters that are not necessarily next to each other. In this paper, we evaluate an handwritten word recognition method based on original open-bigrams representation. We trained Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) to predict open-bigrams rather than characters, and we show that such models are able to learn the long-range, complicated and intertwined dependencies in the input signal, necessary to the prediction. For decoding, we decomposed each word of a large vocabulary into the set of constituent bigrams, and apply a simple cosine similarity measure between this representation and the bagged RNN prediction to retrieve the vocabulary word. We compare this method to standard word recognition techniques based on sequential character recognition. Experiments are carried out on two public databases of handwritten words (Rimes and IAM), an the results with our bigram decoder are comparable to more conventional decoding methods based on sequences of letters.
1 INTRODUCTION
Taking inspiration in Biology is sometimes very efficient. For example, deep neural networks (NN) – which are outperforming all other methods (including support vector machines, SVM) in image recognition – are based on a series of several (usually 5 to 15) neurons layers, each layer involving sparsity in the activation pattern (a biological trait of the cortical map). The analogy continues with the modeling of the cortex as a hierarchy of cortical maps. Thanks to the analysis of reaction time in cognitive psychology experiments, the minimal number of cortical maps involved in a cognitive process is estimated to about ten, the same order of magnitude as the number of layers in deep neural networks for computer vision tasks. In the case of handwritten word recognition, Dehaene et al. have proposed a biologically plausible model of the cortical organization of reading (Dehaene et al., 2005) that assumes seven successive steps of increasing complexity, from the retinal ganglion cells to a cortical map of the orthographic word forms (Figure 1). One of the most recent successes of experimental psychology was the demonstration that human visual word recognition uses an explicit representation of letter position order based on letter pairs: the open-bigram coding (Whitney et al., 2012; Gomez et al., 2008; Grainger & Van Heuven, 2003; Glotin et al., 2010; Dufau, 2008).
As demonstrated in (Touzet et al., 2014), open-bigrams (OB) allow an over-coding of the orthographic form of words that facilitates recognition. OB coding favors same length words (i.e., neighbors of similar lengths). In the context of learning to read, the existence of the OB layer just before the orthographic word representation has been used to explain the lack of efficiency of whole language method (today banned from reading teaching) compared to the phonics method which explic-
itly supervises the organization of the OB map (with syllables), where the global method does not (Figure 1).
Since cognitive psychology has demonstrated the existence of the OB layer, the hypothesis has been put forward (Touzet et al., 2014) that the orthographic representation of words may have evolved in order to take into account the topology of the OB space, instead of the topology of the single letter space. Our goal here is to test this hypothesis, comparing OB vs sequential character recognition for word recognition. A state-of-the-art decoder based on a Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) is used on two public databases of handwritten words (Rimes and IAM).
The remaining of this paper will be divided as follows. In Section 2, we present related methods for handwritten word recognition. Then, we describe the open-bigram representation of words and the proposed decoder in Section 3. The experimental setup, including the data and the bigram prediction model, is explained in Section 4. Finally, we present our results in Section 5, before concluding in Section 6.
2 RELATED WORK
In this section, we give a brief overview of existing techniques for handwritten word recognition. Historically, the methods may be divided in three broad categories. The first approach is whole word recognition, where the image of the full word is directly classified into word classes, without relying on the character level (e.g. in (Parisse, 1996; Madhvanath & Govindaraju, 1996)). In the second method, the word image is segmented into parts of characters (stokes or graphemes). The segments are grouped and scored, and character sequences are obtained with a graph search (e.g. in (Bengio et al., 1995)) or with hidden Markov models (HMMs, e.g. in (Knerr et al., 1998)). The last method, most popular nowadays, is a segmentation-free approach. The goal is to predict a character sequence from the image without segmenting it first. The techniques include scanning a sliding window to extract features used in an HMM (e.g. in (Kaltenmeier et al., 1993)), or to feed the image to a neural network able to output sequences of character predictions (e.g. SDNNs (LeCun et al., 1998) or MDLSTM-RNN (Graves & Schmidhuber, 2009)).
More recently, different approaches have been proposed to recognize words using character bigrams, and therefore closer to the method we propose in this paper. Jaderberg et al. (2014) propose
to predict both the characters and ngrams of characters with two distinct convolutional neural networks (CNNs) to recognize text in natural images. Their approach includes a conditional random field as decoder. Similarly, Poznanski & Wolf (2016) train a CNN with a cross-entropy loss to detect common unigrams, bigrams or trigrams of character in a handwritten word image. The output of the network is matched against the lexicon using canonical correlation analysis. Almazán et al. (2014) use Fisher vectors from images and pyramidal character histograms, to learn a feature space shared by the word images and labels, for word spotting, also using canonical correlation analysis.
3 PROPOSED METHOD
3.1 AN OPEN-BIGRAM REPRESENTATION OF WORDS
The letter bigrams of a word w is the set of pairs of consecutive letters. The open-bigram of order d is the set of pairs of letters separated by d other letters in the word, which we call Bd(w):
Bd(w) = {wiwi+d : i ∈ {1 . . . |w| − d}}. (1)
The usual bigrams are open-bigrams of order 1. By extension, we call B0(w) the set of letters in the word w. For example, for word word, we have:
B1(word) = {or, rd, wo} ; B2(word) = {od, wr} ; B3(word) = {wd}.
The general open-bigram representation of a word is the union of
Bd1,...,dn(w) = Bd1(w) ∪ . . . ∪ Bdn(w). (2)
For example, B1,2,3(word) = {od, or, rd, wd, wo, wr}. We extend B into B′ by including special bigrams for the letters at the beginning and end of a word:
B′(w) = B(w) ∪ { w0, w|w| }. (3) So, for example,
B′1,2,3(word) = { w, d , od, or, rd, wd, wo, wr}. (4)
In this paper, we will call B the set of all bigrams, and W the set of all words. We will represent a word of the vocabulary w ∈W as a normalized binary vector vw∈W ∈ <|B|
vw = [δ(b ∈ B(w))]b∈B√
|B(w)| , (5)
i.e. the vector with 0 everywhere and 1/ √ |B(w)| at indices corresponding to bigrams of the word. The stacking of the vector representation of all the words in the vocabulary yields the vocabulary matrix V ∈ <|W |×|B|. Note that in this representation, the bigrams form an unordered set. We do not know: (i) where the bigrams are, (ii) what is the order of a given bigram, (iii) how many times it occurs. The goal is to build a word recognition decoder in the bigram space.
3.2 AN OPEN-BIGRAM DECODER
While the trivial representation of a word is an ordered sequence of letters, the order in the bigram space is locally embedded in the bigram representation. Most state-of-the-art word recognition systems recognize sequences of letters, and organize the vocabulary for a constrained search as directed graphs, such as prefix trees, or Finite-State Transducers. On the other hand, we can interpret the bigram representation as encoding directed edges in a graph, although we will not explicitly build such a graph for decoding.
On Figure 2, we show the graph for a representation of the word into a sequence of letters. Gray edges show the potential risk of a misrecognition in the letter sequences. On Figure 2(b), we display the conceptual representation of bigrams as edges. We observe that a global order of letters can emerge from the local representation. Moreover, the constituent information of a word in the
bigram space is redundant, potentially making this representation more robust to mispredictions of the optical model.
The optical model is the system which provides the predictions of bigrams from the image (or, in the classical approach sequences of character predictions). That is, it provides a confidence measure that each bigram b is present in image x: 0 ≤ pb(x) ≤ 1. This is transformed into a vector in the bigram space:
qx = [pb(x)]b∈B√∑
b p 2 b(x)
. (6)
For decoding, we chose the very simple cosine similarity between the query (qx) and a vocabulary word (vw). Since we normalized both vectors, this is simply the dot product:
d(qx,vw) = v T wqx, (7)
so the similarity with all words of the vocabulary can be computed with a matrix-vector product:
DV (x) = V Tqx. (8)
The recognized word is the one with maximum similarity with the query:
w∗ = argmaxDV (x) = argmax w ∑ b∈B(w) pb(x)√ |B(w)| ∑ b p 2 b(x) . (9)
We carried out a few preliminary experiments to justify the open-bigram decoder. First, we considered the famous sentence with mixed up letters:
“aoccdrnig to a rscheearch at cmabrigde uinervtisy it deos not mttaer in waht oredr the ltteers in a wrod are the olny iprmoatnt tihng is taht the frist and lsat ltteers be at the rghit pclae the rset can be a toatl mses and you can sitll raed it wouthit porbelm tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef but the wrod as a wlohe”.
Although the origin and validity of this statement when letters are put in the right order has been discussed 1, it is true that most of us can read it without trouble. For each word of more than one letter in this sentence, we computed the open-bigram representation (d = 0..3), and replaced it with the word having the highest cosine similarity in the English vocabulary described in the next section. The result was:
“according to a researcher at abridged university it does not matter in what ordered the letters in a word are the only important thing is that the first and last letters be at the right place the rest can be a total messes and you can still read it outwith problem this is because the human mind does not read every letter by itself but the word as a whole”.
Note that the word “cambridge” was not in the vocabulary. Although the task in this paper is not to recognize mixed up words, it shows the ability of our decoder to perform a reading task that we naturally do.
1http://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
On Figure 3, we show the English vocabulary in bigram space (d = 1..3), reduced to two dimensions with t-SNE (Van der Maaten & Hinton, 2008). We observe that words which are close in the bigram space also have a close orthographic form.
4 EXPERIMENTAL SETUP
4.1 DATA PREPARATION
We carried out the experiments on two public handwritten word databases: Rimes (Augustin et al., 2006) (French), and IAM (Marti & Bunke, 2002) (English). We simplified the problem by limiting ourselves to words of at least two lowercase characters (a to z). This selection removed approximately 30% of the words. The number of words and bigrams of different orders in the different sets are reported on Table 4, in Appendix A.1.
We applied deslanting (Buse et al., 1997), contrast enhancement, and padded the images with 10px of white pixels to account for empty context on the left and right of words. From the preprocessed images, we extracted sequences of feature vectors with a sliding window of width 3px. The features are geometrical and statistical features described in (Bianne et al., 2011), which give state-of-the-art results in handwritten text line recognition (Bluche et al., 2014).
We downloaded word frequency lists for French and English2. These lists were built from film subtitles 3 written by many contributors, and they contain many misspellings. We removed the misspelled words using GNU Aspell (Atkinson).
We selected 50,000 words for each language. They are the most frequent words (length ≥ 2) and made only of lowercase characters between a and z, making sure to also include all the words of the database. For example, the 50,000 most frequent French words fulfilling these condition miss about 200 words of the Rimes database, so we selected the most frequent 49,800 and added the missing 200. Note that most of the words that were removed from the dataset are not shorter words, but words with special or upper case characters. The distribution of lengths of filtered-out words is shown on Figure 5 in Appendix A.1.
4.2 RECOGNITION OF OPEN-BIGRAMS WITH RECURRENT NEURAL NETWORKS (RNNS)
To predict bigrams, we chose Bidirectional Long Short-Term Memory RNNs (BLSTM-RNNs) for their ability to consider the whole sequence of input vectors to make predictions. We trained one RNN for each order-d bigram, with the Connectionist Temporal Classification (CTC (Graves et al., 2006)) criterion. The CTC framework defines a sequence labeling problem, with an output sequence of labels, of smaller length than the input sequence of observations.
2http://invokeit.wordpress.com/frequency-word-lists/. 3http://opensubtitles.org
We built the target sequences for training as sequences of bigrams, ordered according to the first letter of the bigram. For example, for d = 2, the target for example is ea-xm-ap-ml-pe. The CTC training criterion optimizes the Negative Log-Likelihood (NLL) of the correct label sequence. We set the learning rate to 0.001, and stopped the training when the NLL on the validation set did not decrease for 20 epochs. We kept the network yielding the best NLL on the validation set.
We trained one RNN for each order d = 0 to 3, including the special bigrams for word extremities or not. We will refer to each of these RNNs with rnnd for order d (rnnd′ when extremities are included). The architecture of the networks is described in Appendix A.3. These RNNs are trained to predict sequences of fixed order bigrams. Here, we are interested in a word representation as a bag of bigrams, which does not carry any information about the sequence in which the bigrams appear, the number of times each bigram appears, or the order of each individual bigram. That is, we are interested in a decoder which considers an unordered set of bigrams predictions across bigram orders.
We forget the temporal aspect of bigram predictions by taking the maximum value of a given bigram prediction by the RNN (where rnnd(x, t) if the output of the RNN for order d, input image x at timestep t):
pd,b(x) = max t rnnd(x, t), (10)
and we forget the bigram order by taking the maximum output across different values of d:
pb(x) = max d max t rnnd(x, t). (11)
It would have been more satisfying for this experiment to train an optical model to predict a set of bigrams for all orders. However, this work is focused on the decoder. Moreover, even the simpler task of predicting a sequence of bigrams of fixed order is challenging (the sequence error rates of these networks are detailed in Appendix B.2). On Figure 4, we show the hypothetical context needed to make two consecutive predictions, for bigram order d = 0..3. RNNs are popular for handwriting recognition, and can consider a context size of variable length – but still local – to predict characters (d = 0).
For d = 1, the required context is still local (and would span two consecutive characters), but overlap, because each character is involved in two bigrams. For d > 1, the context is even split into two areas (covering the involved characters) that might be far apart depending on d. Contexts for different predictions are entangled: the whole area between two characters forming a bigram is not relevant for this bigram (and might be of varying size), but will be important to predict other bigrams. It means that the RNN will have to remember a character observation for some time, until it sees the second character of the bigram, while ignoring the area in between for this bigram prediction, but remembering it since it will be useful in order to predict other bigrams. The number of classes for bigrams is also 26 times larger than the number of characters, making the classification problem harder, and the number of examples per class in training smaller.
5 RESULTS
In this paper, we focused on a subset of Rimes and IAM word databases, which makes the comparison with published results difficult. Instead, we compared the bigram decoder approach to decoding with standard models, consisting of a beam search with Viterbi algorithm in the lexicon. However, these standard models yield state-of-the art results on the reference task for the two considered database (Bluche et al., 2014).
5.1 BASELINE SYSTEMS BASED ON HMMS AND VITERBI DECODING
We built several models and used the same vocabulary as for the bigram decoder, and no language model (all words have the same prior probability). These baseline systems are based on HMMs, with emission models made either of Gaussian mixtures (GMM/HMM), Multi-Layer Perceptrons (MLP/HMM) or Recurrent Neural Networks (rnn0/HMM). They are almost identical to those presented in a previous work (Bluche et al., 2014), where a comparison is made with state-of-the-art systems for handwritten text line recognition. More details about these models and their training procedure are presented in Appendix A.2.
On Table 1, we report the percentages of word errors on the validation and test sets of Rimes and IAM. The best word error rates are around 10% (17.5% on the test set of IAM), and constitute the baseline performance to which the bigram approach is to be compared.
5.2 MEASURING THE QUALITY OF BIGRAM PREDICTIONS
Since we keep a confidence value for all bigrams in the prediction vector, rather than using a binary vector (cf. Eq. 6), we modified the formulation of precision and recall. A bigram b ∈ B(w) is correctly retrieved with confidence pb(x), and missed with confidence (1 − pb(x)). Similarly, a bigram not in the representation B(w) of word w is falsely recognized with confidence pb(x), and correctly ignored with confidence (1 − pb(x)). It gives us the following expressions for precision and recall
precision =
∑ (x,w) ∑ b∈B(w) pb(x)∑
x ∑ b′∈B pb(x) , recall =
∑ (x,w) ∑ b∈B(w) pb(x)∑
w∈W |B(w)| , (12)
which are the usual ones when pb(x) ∈ {0, 1}. The F-measure is calculated from precision and recall with the usual formula.
The results for all RNNs, and for the combination of orders, are reported on Table 2. We observe that the precision and recall results are correlated to the performance in terms of edit distance or sequence error rates. Namely, they decrease as the bigram order increases, which is not surprising, given that higher order bigrams are more difficult to recognize with these sequence models. We also see that including the special bigrams for word beginnings and endings generally improves the results. This is not surprising either: the RNNs are good at recognizing them.
Despite this performance decrease, the precision remains above 70%, which limits the amount of noise that will be included in the bigram representation for recognition. Combining the recognition across orders, we obtain a precision of around 84% on Rimes and 80% on IAM. The recall tends to be higher than the precision, staying around or above 80% in all configurations. Across orders, the
recall is above 88% on Rimes and 86% on IAM. The high recall will limit the amount of missing information in the bigram representation.
Overall, the F-measure for bigram recognition is above 80%, which is a good starting point, given that (i) the vocabulary used in decoding will add constraints and may help recovering from some mistakes in the bigram recognition, and (ii) the redundancy and order encoded in the bigram may limit the impact of misrecognitions.
5.3 WORD RECOGNITION USING BIGRAM PREDICTIONS
On Table 3, we report the results of bigram decoding. For each word image in the validation and test sets, we computed the bigram predictions with the RNNs described above. We combined the different orders as explained previously, and either added the special bigrams for word boundaries and/or the single character predictions or not. We computed the cosine similarity to the bigram decomposition of all words in the vocabularies in the same representation space (i.e. same orders, and same choices for the inclusion of special bigrams and single characters) by computing the product of the vocabulary matrix V by the recognition vector. We counted the number of times the correct word was not the most similar one.
We see that adding the special bigrams for word boundaries improves the results, especially when single characters are not included in the representation. A possible explanation, besides the fact that they tend to be recognized more easily, could be that they provide a very useful information to disambiguate words having a similar bigram representation (e.g. them and theme). Adding single characters also improves the performance of the decoder, especially when the boundary bigrams are not included in the representation. The gain obtained with the single characters is about the same – sometimes a little better – as the gain with boundaries. It might be due to the much better recognition of the RNN for single characters (precision and recall over 90%), as well as the added redundancy and complementary information provided. The results of decoding with different combinations of orders are presented in the appendices in Table 7. They confirm those observations. The best performance is achieved with both single characters and word boundaries, although the gain compared to adding only one of them is slight. The error rates are competitive or better than the best error rates obtained by classical character sequence modeling and Viterbi decoding.
6 CONCLUSION
State-of-the-art systems, as well as most of the systems for handwritten word recognition found in the literature, either try to model words as a whole, or as a sequence of characters. The latter, which currently gives the best results, is widely adopted by the community, and benefits from a lot of attention. In this paper, we have proposed a simple alternative model, inspired by the recent findings in cognitive neurosciences research on reading.
We focused on the representation of words in the open-bigram space and built an handwritten word recognition system operating in that space. We were interested in observing how a simple decoding scheme, based on a mere cosine similarity measure in the bigram space, compared to traditional methods. The main apparent difficulty arises from the fact that the global ordering of characters and the distance between bigram constituents are lost in this representation.
The qualitative results presented in the first section showed that the envisioned approach was viable. With the letter reordering example, we have seen that the correct orthographic form of words can be retrieved with a limited and local knowledge of character orders. Moreover, we validated that words
that are close in orthographic form are also close in the bigram space. Thus, we demonstrated that the open-bigram representation shows interesting and competitive metric properties for the word recognition. Current work consists in learning most discriminant open-bigram at different order, possibly higher than three according to the length of the word and its similarity to others.
ACKNOWLEDGMENTS
This work was conducted in COGNILEGO project 2012-15, supported by the French Research Agency under the contract ANR 2010-CORD-013 http://cognilego.univ-tln.fr.
A DATA AND MODELS
A.2 BASELINE SYSTEMS
We built several models and used the same vocabulary as for the bigram decoder, and no language model (all words have the same prior probability). The first one is a Hidden Markov Model, with 5 (Rimes) or 6 (IAM) states per characters, and an emission model based on Gaussiam Mixture Models. This system is trained with the Maximum Likelihood criterion, following the usual ExpectationMaximization procedure. At each iteration, the number of Gaussians is increased, until no improvement is observed on the validation set. The forced alignments with the GMM/HMM system are used to build a labeled dataset for training a Multi-Layer Perceptron (MLP) with 4 hidden layers of 1,024 sigmoid units. We optimized the cross-entropy criterion to train the network to predict the HMM states from the concatenation of 11 input frames, with a learning rate of 0.008. The learning rate was halved when the relative improvement was smaller than 1% from one epoch to the next. The MLP is integrated in the hybrid NN/HMM scheme by dividing the predicted state posteriors p(s|x) by the state priors p(s), estimated from the forced alignments. It is then further trained with a sequence-discriminative criterion: state-level Minimum Bayes Risk (Kingsbury, 2009) (sMBR) for 5 epochs. Finally, we also performed the sequential decoding with vocabulary constraints using rnn0. The input features for all systems are the same, described in 4.1. The results of similar
systems for handwritten text line recognition can for example be found in a previous work (Bluche et al., 2014), where a comparison is made with state-of-the-art systems.
A.3 RECURRENT NEURAL NETWORK
The RNNs, depicted on Figure 6, have seven hidden layers, alternating Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) recurrent layers in both direction and feed-forward layers. The first LSTM layers have 100 hidden LSTM units. The BLSTM outputs of two consecutive timesteps of both directions are fed to a feed-forward layer with 100 nodes, halving the size of the sequence. The next LSTM layers have 150 units, and are connected to a feed-forward layer with 150 units. The last two LSTM layers and feed-forward layer have 200 hidden units.
The inputs of the networks are sequences of geometrical and statistical features (described in (Bianne et al., 2011)), extracted with a sliding window of width 3px. The outputs are sequences of openbigram predictions.
The networks are trained with stochastic gradient descent to minimize the Connectionist Temporal Classification (CTC (Graves et al., 2006)) criterion, i.e. the Negative Log-Likelihood (NLL) of the correct label sequence. We set the learning rate to 0.001, and stopped the training when the NLL on the validation set did not decrease for 20 epochs, and kept the network yielding the best NLL on the validation set.
B RESULTS
B.1 CORRELATION BETWEEN CHARACTER EDIT DISTANCES AND OB COSINE SIMILARITIES
On Figure 7, we randomly selected pairs of words, and pairs of words with high cosine similarity in the French and English dictionary, and plotted their cosine similarity in the bigram space against
the edit distance between the two words, normalized by the length of the longest word. We note that words with high cosine similarity also have short edit distance, supporting the idea that the bigram representation encode some global letter order, and therefore might favor word recognition.
B.2 EVALUATION OF PERFORMANCE OF RNN ALONE
We trained RNNs to predict sequences of bigrams of a given order. Their performance to accomplish this task can be measured, on the validation set, with the edit distance (edit.dist) between the recognized sequence and the true sequence, and with the percentage of sequences with at least one error: the Sequence Error Rate (SER).
The results are reported on Table 6. We observe that the performance decreases as d increase. Yet the errors remain in a reasonable range compared to the simple case d = 0, and one should keep in mind that a small degradation should be expected from the larger number of classes and smaller number of training examples per class.
We have shown that RNNs can perform the apparently difficult task of recognizing sequences of open-bigrams. Thus, we may use the RNN bigram predictions in our decoder, rather than building bigram predictions from letter ones, which would have been less satisfying considering the supposed interest of the redundancy of the bigram representation that allows error corrections.
B.3 EVALUATION OF PERFORMANCE OF THE DECODER
In Table 7, we report the results of the cosine decoder applied to the outputs of OB RNNs with different bigram orders.
In Table 8, we evaluate the cosine decoder by applying it to the ground-truth OB decomposition with varying order of the validation sets, i.e. the performance assuming the RNN optical models are perfect.
B.4 ERROR ANALYSIS
B.4.1 COMPARISON OF ERRORS OF THE SEQUENTIAL AND OPEN-BIGRAM MODELS

This paper uses an LSTM model to predict what it calls "open bigrams" (bigrams of characters that may or may not have letters inbetween) from handwriting data. These open bigrams are subsequently used to predict the written word in a decoding step. The experiments indicate that the system does slightly better than a baseline model that uses Viterbi decoding. I have some major concerns about this paper:
- I find the "cortical inspired" claim troublesome. If anything, it is psychology/cognitive science inspired, in the sense that open bigrams appear to help for word recognition (Touzet et al. 2014). But the implied cortical characteristics, implicitly referred to e.g. by pointing to analogies between deep neural nets for object recognition and in that case the visual cortex, is unfounded. Is there any direct evidence from neuroscience that open-bigrams constitute a wholly separate layer in the cortex for a handwriting recognition task? Dehaene's work is a proposal, so you'll need to describe more "findings in cognitive neurosciences [sic] research on reading" (p. 8) to substantiate those claims. I am further worried by the fact that the authors seem to think that "deep neural networks are based on a series of about five pairs of neurons [sic] layers". Unless I misunderstand something, you are specifically referring to Krizhevsky's AlexNet here (which you should probably have cited there)? I hope you don't mean to imply that all deep neural nets need five layers. It is also not true that ten is "quite close to the number of layers of an efficient deep NN" -- what network? what task? etc.
- The model is not clearly explained. There is a short paragraph in Appendix A.3. that roughly describes the setup, but this does not include e.g. the objective function, or answer why the network output is only considered each two consecutive time steps, rather than at each time step (or so it seems?). This is probably because the paper argues that it "is focused on the decoder" (p. 6), rather than on the whole problem. I find this problematic, because in that case we're effectively measuring how easy it is to reconstruct a word from its open bigrams, which has very little to do with handwriting recognition (it could have been evaluated on any text corpus). In fact, as the example on page 4 shows, handwriting is not necessary to illustrate the open bigram hypothesis. Which leads me to wonder why these particular tasks were chosen, if we are only interested in the decoding mechanism?
- The comparison is not really fair. The Viterbi decoder only has access to unigrams, as far as I can tell. The only model that does better than that baseline has access to a lot more information, and does not do that much better. Did the Viterbi model have access to the word boundary information (at one point rather confusingly called "extremities") that pushed the open bigram model over the edge in terms of performance? Why is there no comparison to e.g. rnn_0,1' (unigram+bigram+boundary markers)? The dataset also appears to be biased in favor of the proposed approach (longer words, only ). I am not convinced that this paper really shows that open bigrams help.
I very much like the idea of the paper, but I am simply not convinced by its claims.
Minor points:
- There are quite a few typos. Just a sample: "independant" (Fig.1), "we evaluate an handwritten", ", hand written words [..], an the results", "their approach include", "the letter bigrams of a word w is", "for the two considered database"
- Wouldn't it be easy to add how many times a bigram occurs, which would improve the decoding process? You can just normalize over the full counts instead of the binary occurrence counts.
- The results in Table 5 are the same (but different precision) as the results in Table 2, except that edit distance and SER are added, this is confusing.