HOLSTEP: A MACHINE LEARNING DATASET FOR HIGHER-ORDER LOGIC THEOREM PROVING
Large computer-understandable proofs consist of millions of intermediate logical steps. The vast majority of such steps originate from manually selected and manually guided heuristics applied to intermediate goals. So far, machine learning has generally not been used to filter or generate these steps. In this paper, we introduce a new dataset based on Higher-Order Logic (HOL) proofs, for the purpose of developing new machine learning-based theorem-proving strategies. We make this dataset publicly available under the BSD license. We propose various machine learning tasks that can be performed on this dataset, and discuss their significance for theorem proving. We also benchmark a set of simple baseline machine learning models suited for the tasks (including logistic regression, convolutional neural networks and recurrent neural networks). The results of our baseline models show the promise of applying machine learning to HOL theorem proving.
1 INTRODUCTION
As the usability of interactive theorem proving (ITP) systems (Harrison et al., 2014) grows, its use becomes a more common way of establishing the correctness of software as well as mathematical proofs. Today, ITPs are used for software certification projects ranging from compilers (Leroy, 2009) and operating system components (Chen et al., 2016; Klein et al., 2014), to establishing the absolute correctness of large proofs in mathematics such as the Kepler conjecture (Hales et al., 2015) and the Feit-Thomson Theorem (Gonthier et al., 2013).
For results of such significance to be possible, the theorem libraries of these ITPs must contain all necessary basic mathematical properties, accompanied with formal proofs. This means that the size of many ITP libraries can be measured in dozens of thousands of theorems (Grabowski et al., 2010; Blanchette et al., 2015) and billions of individual proof steps. While the general direction of the proofs is specified by humans (by providing the goal to prove, specifying intermediate steps, or applying certain automated tactics), the majority of such proof steps are actually found by automated reasoning-based proof search (Kaliszyk & Urban, 2015b), with very little application of machine learning techniques so far.
At the same time, fast progress has been unfolding in machine learning applied to tasks that involve logical inference, such as natural language question answering (Sukhbaatar et al., 2015), knowledge base completion (Socher et al., 2013a), automated translation (Wu et al., 2016), and premise selection in the context of theorem proving (Alemi et al., 2016). Deep learning in particular has proven to be a powerful tool for embedding semantic meaning and logical relationships into geometric spaces, specifically via models such as convolutional neural networks, recurrent neural networks, and tree-recursive neural networks. These advances strongly suggest that deep learning may have become mature enough to yield significant advances in automated theorem proving. Remarkably, it has recently become possible to build a system, AlphaGo (Silver et al., 2016), blending classical AI techniques such as Monte-Carlo tree search and modern deep learning techniques, capable of playing the game of Go at super-human levels. We should note that theorem proving and Go playing are conceptually related, since both consist in searching for specific nodes in trees of states with extremely large arity and relatively large depth, which involves node evaluation decision (how valuable is this state?) and policy decisions (which node should be expanded next?). The success of AlphaGo can thus serve as encouragement on the road to building deep learning-augmented theorem
provers that would blend classical techniques developed over the past few decades with the latest machine learning advances.
Fast progress in specific machine learning verticals has occasionally been achieved thanks to the release of specialized datasets (often with associated competitions, e.g. the ImageNet dataset for large-scale image classification (Deng et al., 2009)) serving as an experimental testbed and public benchmark of current progress, thus focusing the efforts of the research community. We hope that releasing a theorem proving dataset suited for specific machine learning tasks can serve the same purpose in the vertical of applying machine learning to theorem proving.
1.1 CONTRIBUTION AND OVERVIEW
First, we develop a dataset for machine learning based on the proof steps used in a large interactive proof section 2. We focus on the HOL Light (Harrison, 2009) ITP, its multivariate analysis library (Harrison, 2013), as well as the formal proof of the Kepler conjecture (Hales et al., 2010). These formalizations constitute a diverse proof dataset containing basic mathematics, analysis, trigonometry, as well as reasoning about data structures such as graphs. Furthermore these formal proof developments have been used as benchmarks for automated reasoning techniques (Kaliszyk & Urban, 2014).
The dataset consists of 2,013,046 training examples and 196,030 testing examples that originate from 11,400 proofs. Precisely half of the examples are statements that were useful in the currently proven conjectures and half are steps that have been derived either manually or as part of the automated proof search but were not necessary in the final proofs. The dataset contains only proofs of non-trivial theorems, that also do not focus on computation but rather on actual theorem proving. For each proof, the conjecture that is being proven as well as its dependencies (axioms) and may be exploited in machine learning tasks. Furthermore, for each statement both its human-readable (pretty-printed) statement and a tokenization designed to make machine learning tasks more manageable are included.
Next, in section 3 we discuss the proof step classification tasks that can be attempted using the dataset, and we discuss the usefulness of these tasks in interactive and automated theorem proving. These tasks include unconditioned classification (without access to conjectures and dependencies) and conjecture-conditioned classification (with access to the conjecture) of proof steps as being useful or not in a proof. We outline the use of such classification capabilities for search space pruning and internal guidance, as well as for generation of intermediate steps or possible new lemma statements.
Finally, in section 4 we propose three baseline models for the proof step classification tasks, and we experimentally evaluate the models on the data in section 5. The models considered include both a relatively simple regression model, as well as deep learning models based on convolutional and recurrent neural networks.
1.2 RELATED WORK
The use of machine learning in interactive and automated theorem proving has so far focused on three tasks: premise selection, strategy selection, and internal guidance. We shortly explain these.
Given a large library of proven facts and a user given conjecture, the multi-label classification problem of selecting the facts that are most likely to lead to a successful proof of the conjecture has been usually called relevance filtering or premise selection (Alama et al., 2014). This is crucial for the efficiency of modern automation techniques for ITPs (Blanchette et al., 2016), which today can usually solve 40–50% of the conjectures in theorem proving libraries. Similarly most competitive ATPs today (Sutcliffe, 2016) implement the SInE classifier (Hoder & Voronkov, 2011).
A second theorem proving task where machine learning has been of importance is strategy selection. With the development of automated theorem provers came many parameters that control their execution. In fact, modern ATPs, such as E (Schulz, 2013) and Vampire (Kovács & Voronkov, 2013), include complete strategy description languages that allow a user to specify the orderings, weighting functions, literal selection strategies, etc. Rather than optimizing the search strategy globally, one
can choose the strategy based on the currently considered problem. For this some frameworks use machine learning (Bridge et al., 2014; Kühlwein & Urban, 2015).
Finally, an automated theorem prover may use machine learning for choosing the actual inference steps. It has been shown to significantly reduce the proof search in first-order tableaux by the selection of extension steps to use (Urban et al., 2011), and has been also successfully applied in monomorphic higher-order logic proving (Färber & Brown, 2016). Data/proof mining has also been applied on the level of interactive theorem proving tactics (Duncan, 2007) to extract and reuse repeating patterns.
2 DATASET EXTRACTION
We focus on the HOL Light theorem prover for two reasons. First, it follows the LCF approach1). This means that complicated inferences are reduced to the most primitive ones and the data extraction related modifications can be restricted the primitive inferences and it is relatively easy to extract proof steps at an arbitrary selected level of granularity. Second, HOL Light implements higher-order logic (Church, 1940) as its foundation, which on the one hand is powerful enough to encode most of today’s formal proofs, and on the other hand allows for an easy integration of many powerful automation mechanisms (Baader & Nipkow, 1998; Paulson, 1999).
When selecting the theorems to record, we choose an intermediate approach between HOL Light ProofRecording (Obua & Skalberg, 2006) and the HOL/Import one (Kaliszyk & Krauss, 2013). The theorems that are derived by most common proof functions are extracted by patching these functions like in the former approach, and the remaining theorems are extracted from the underlying OCaml programming language interpreter. In certain cases decision procedures derive theorems to be reused in subsequent invocations. We detect such values by looking at theorems used across proof blocks and avoid extracting such reused unrelated subproofs.
All kernel-level inferences are recorded together with their respective arguments in a trace file. The trace is processed offline to extract the dependencies of the facts, detect used proof boundaries, mark the used and unused steps, and mark the training and testing examples. Only proofs that have sufficiently many used and unused steps are considered useful for the dataset. The annotated proof trace is processed again by a HOL kernel saving the actual training and testing examples originating from non-trivial reasoning steps. Training and testing examples are grouped by proof: for each proof the conjecture (statement that is finally proved), the dependencies of the theorem are constant, and a list of used and not used intermediate statements is provided. This means that the conjectures used in the training and testing sets are normally disjoint.
For each statement, whether it is the conjecture, a proof dependency, or an intermediate statement, both a fully parenthesised HOL Light human-like printout is provided, as well as a predefined tokenization. The standard HOL Light printer uses parentheses and operator priorities to make its notations somewhat similar to textbook-style mathematics, while at the same time preserving the complete unambiguity of the order of applications (this is particularly visible for associative operators). The tokenization that we propose attempts to reduce the number of parentheses. To do this we compute the maximum number of arguments that each symbol needs to be applied to, and only mark partial application. This means that fully applied functions (more than 90% of the applications) do not require neither application operators nor parentheses. Top-level universal quantifications are eliminated, bound variables are represented by their de Bruijn indices (the distance from the corresponding abstraction in the parse tree of the term) and free variables are renamed canonically. Since the Hindley-Milner type inference Hindley (1969) mechanisms will be sufficient to reconstruct the most-general types of the expressions well enough for automated-reasoning techniques Kaliszyk et al. (2015) we erase all type information. Table 1 presents some dataset statistics. The dataset, the description of the used format, the scripts used to generate it and baseline models code are available:
http://cl-informatik.uibk.ac.at/cek/holstep/
1LCF approach is a software architecture for implementing theorem provers which uses a strongly typed programming language with abstract datatypes (such as OCaml in the case of HOL Light) to separate the small trusted core, called the kernel, which verifies the primitive inferences from user code which allows the user to arbitrarily extend the system in a safe manner. For more details see (Gordon et al., 1979).
3 MACHINE LEARNING TASKS
3.1 TASKS DESCRIPTION
This dataset makes possible several tasks well-suited for machine learning most of which are highly relevant for theorem proving:
• Predicting whether a statement is useful in the proof of a given conjecture;
• Predicting the dependencies of a proof statement (premise selection);
• Predicting whether a statement is an important one (human named);
• Predicting which conjecture a particular intermediate statement originates from;
• Predicting the name given to a statement;
• Generating intermediate statements useful in the proof of a given conjecture;
• Generating the conjecture the current proof will lead to.
In what follows we focus on the first task: classifying proof step statements as being useful or not in the context of a given proof. This task may be further specialized into two different tasks:
• Unconditioned classification of proof steps: determining how likely a given proof is to be useful for the proof it occurred in, based solely on the content of statement (i.e. by only providing the model with the step statement itself, absent any context).
• Conditioned classification of proof steps: determining how likely a given proof is to be useful for the proof it occurred in, with “conditioning” on the conjecture statement that the proof was aiming to attain, i.e. by providing the model with both the step statement and the conjecture statement).
In the dataset, for every proof we provide the same number of useful and non-useful steps. As such, the proof step classification problem is a balanced two-class classification problem, where a random baseline would yield an accuracy of 0.5.
3.2 RELEVANCE TO INTERACTIVE AND AUTOMATED THEOREM PROVING
In the interaction with an interactive theorem prover, the tasks that require most human time are: the search for good intermediate steps; the search for automation techniques able to justify the individual steps, and searching theorem proving libraries for the necessary simpler facts. These three problems directly correspond to the machine learning tasks proposed in the previous subsection. Being able to predict the usefulness of a statement will significantly improve many automation techniques. The generation of good intermediate lemmas or intermediate steps can improve level of granularity of the proof steps. Understanding the correspondence between statements and their names can allow users to search for statements in the libraries more efficiently (Aspinall & Kaliszyk, 2016). Premise selection and filtering are already used in many theorem proving systems, and generation of succeeding steps corresponds to conjecturing and theory exploration.
4 BASELINE MODELS
For each task (conditioned and unconditioned classification), we propose three different deep learning architectures, meant to provide a baseline for the classification performance that can be achieved on this dataset. Our models cover a range of architecture features (from convolutional networks to recurrent networks), aiming at probing what characteristics of the data are the most helpful for usefulness classification.
Our models are implemented in TensorFlow (Abadi et al., 2015) using the Keras framework (Chollet, 2015). Each model was trained on a single Nvidia K80 GPU. Training only takes a few hours per model, which makes running these experiments accessible to most people (they could even be run on a laptop CPU). We are releasing all of our benchmark code as open-source software 2 so as to allow others to reproduce our results and improve upon our models.
4.1 UNCONDITIONED CLASSIFICATION MODELS
Our three models for this task are as follow:
• Logistic regression on top of learned token embeddings. This minimal model aims to determine to which extent simple differences between token distribution between useful and non-useful statements can be used to distinguish them. It provides an absolute floor on the performance achievable on this task.
• 2-layer 1D convolutional neural network (CNN) with global maxpooling for sequence reduction. This model aims to determine the importance of local patterns of tokens.
• 2-layer 1D CNN with LSTM (Hochreiter & Schmidhuber, 1997) sequence reduction. This model aims to determine the importance of order in the features sequences.
See figure 1 for a layer-by-layer description of these models.
4.2 CONDITIONED CLASSIFICATION MODELS
For this task, we use versions of the above models that have two siamese branches (identical branches with shared weights), with one branch processing the proof step statement being considered, and the
2https://github.com/tensorflow/deepmath/tree/master/holstep_baselines
other branch processing the conjecture. Each branch outputs an embedding; these two embeddings (step embedding and conjecture embedding) are then concatenated and the classified by a fullyconnected network. See figure 2 for a layer-by-layer description of these models.
4.3 INPUT STATEMENTS ENCODING
It should be noted that all of our models start with an Embedding layer, mapping tokens or characters in the statements to dense vectors in a low-dimensional space. We consider two possible encodings for presenting the input statements (proof steps and conjectures) to the Embedding layers of our models:
• Character-level encoding of the human-readable versions of the statements, where each character (out of a set of 86 unique characters) in the pretty-printed statements is mapped to a 256-dimensional dense vector. This encoding yields longer statements (training statements are 308 character long on average).
• Token-level encoding of the versions of the statements rendered with our proposed highlevel tokenization scheme. This encoding yields shorter statements (training statements are 60 token long on average), while considerably increasing the size of set of unique tokens (1993 total tokens in the training set).
5 RESULTS
Experimental results are presented in tables 2 and 3, as well as figs. 3 to 6.
5.1 INFLUENCE OF MODEL ARCHITECTURE
Our unconditioned logistic regression model yields an accuracy of 71%, both with character encoding and token encoding (tables 2 and 3). This demonstrates that differences in token or character distributions between useful and non-useful steps alone, absent any context, is sufficient for discriminating between useful and non-useful statements to a reasonable extent. This also demonstrates that the token encoding is not fundamentally more informative than raw character-level statements.
Additionally, our unconditioned 1D CNN model yields an accuracy of 82% to 83%, both with character encoding and token encoding (tables 2 and 3). This demonstrates that patterns of characters or patterns of tokens are considerably more informative than single tokens for the purpose of usefulness classification.
Finally, our unconditioned convolutional-recurrent model does not improve upon the results of the 1D CNN, which indicates that our models are not able to meaningfully leverage order in the feature sequences into which the statements are encoded.
5.2 INFLUENCE OF INPUT ENCODING
For the logistic regression model and the 2-layer 1D CNN model, the choice of input encoding seems to have little impact. For the convolutional-recurrent model, the use of the high-level tokenization seems to cause a large decrease in model performance (figs. 4 and 6). This may be due to the fact that token encoding yields shorter sequences, making the use of a LSTM less relevant.
5.3 INFLUENCE OF CONDITIONING ON THE CONJECTURE
None of our conditioned models appear to be able to improve upon the unconditioned models, which indicates that our architectures are not able to leverage the information provided by the conjecture. The presence of the conditioning does however impact the training profile of our models, in particular by making the 1D CNN model converge faster and overfit significantly quicker (figs. 5 and 6).
6 CONCLUSIONS
Our baseline deep learning models, albeit fairly weak, are still able to predict statement usefulness with a remarkably high accuracy. Such methods already help first-order automated provers (Kaliszyk & Urban, 2015a) and as the branching factor is higher in HOL the predictions are valuable for a number of practical proving applications. This includes making tableaux-based (Paulson, 1999) and superposition-based (Hurd, 2003) internal ITP proof search significantly more efficient in turn
making formalization easier. However, our models do not appear to be able to leverage order in the input sequences, nor conditioning on the conjectures. This is due to the fact that these models are not doing any form of logical reasoning on their input statements; rather they are doing simple pattern matching at the level of n-grams of characters or tokens. This shows the need to focus future efforts on different models that can do reasoning, or alternatively, on systems that blend explicit reasoning (e.g. graph search) with deep learning-based feature learning. A potential new direction would be to leverage the graph structure of HOL statements using e.g. Recursive Neural Tensor Networks (Socher et al., 2013a;b) or other graph-based recursive architectures.
6.1 FUTURE WORK
The dataset focuses on one interactive theorem prover. It would be interesting if the proposed techniques generalize, primarily across ITPs that use the same foundational logic, for example using OpenTheory (Hurd, 2011), and secondarily across fundamentally different ITPs or even ATPs. A significant part of the unused steps originates from trying to fulfill the conditions for rewriting and from calls to intuitionistic tableaux. The main focus is however on the human found proofs so the trained predictions may to an extent mimic the bias on the usefulness in the human proofs. As ATPs are at the moment very week in comparison with human intuition improving this even for the many proofs humans do not find difficult would be an important gain.
Finally, two of the proposed task for the dataset have been premise selection and intermediate sentence generation. It would be interesting to define more ATP-based ways to evaluate the selected premises, as well as to evaluate generated sentences (Kaliszyk et al., 2015). The set is a relatively large one when it comes to proof step classification, however the number of available premises makes the set a medium-sized set for premise selection in comparison with those of the Mizar Mathematical Library or the seL4 development.
ACKNOWLEDGEMENTS
The first author was partly supported by the ERC starting grant 714034.

Use of ML in ITP is an interesting direction of research. Authors consider the problem of predicting whether a given statement would be useful in a proof of a conjecture or not. This is posed as a binary classification task and authors propose a dataset and some deep learning based baselines.
I am not an expert on ITP or theorem proving, so I will present a review from more of a ML perspective. I feel one of the goals of the paper should be to present the problem to a ML audience in a way that is easy for them to grasp. While most of the paper is well written, there are some sections that are not clear (especially section 2):
-	Terms such as LCF, OCaml-top level, deBruijn indices have been used without explaining or any references. These terms might be trivial in ITP literature, but were hard for me to follow.
-	Section 2 describes how the data was splits into train and test set. One thing which is unclear is – can the examples in the train and test set be statements about the same conjecture or are they always statements about different conjectures?
It also unclear how the deep learning models are applied. Let’s consider the leftmost architecture in Figure 1. Each character is embedded into 256-D vector – and processed until the global max-pooling layer. Does this layer take a max along each feature and across all characters in the input?
My another concern is only deep learning methods are presented as baselines. It would be great to compare with standard NLP techniques such as Bag of Words followed by SVM. I am sure these would be outperformed by neural networks, but the numbers would give a sense of how easy/hard the current problem setup is.
Did the authors look at the success and failure cases of the algorithm? Are there any insights that can be drawn from such analysis that can inform design of future models?
Overall I think the research direction of using ML for theorem proving is an interesting one. However, I also feel the paper is quite opaque. Many parts of how the data is constructed is unclear (atleast to someone with little knowledge in ITPs). If authors can revise the text to make it clearer – it would be great. The baseline models seem to perform quite well, however there are no insights into what kind of ability the models are lacking. Authors mention that they are unable to perform logical reasoning – but that’s a very vague statement. Some examples of mistakes might help make the message clearer. Further, since I am not well versed with the ITP literature it’s not possible for me to judge how valuable is this dataset. From the references, it seems like it’s drawn from a set of benchmark conjectures/proofs used in the ITP community – so its possibly a good dataset.
My current rating is a weak reject, but if the authors address my concerns I would change to an accept.


TRANSFER FROM MULTIPLE SOURCES IN THE SAME DOMAIN
Transferring knowledge from prior source tasks in solving a new target task can be useful in several learning applications. The application of transfer poses two serious challenges which have not been adequately addressed. First, the agent should be able to avoid negative transfer, which happens when the transfer hampers or slows down the learning instead of helping it. Second, the agent should be able to selectively transfer, which is the ability to select and transfer from different and multiple source tasks for different parts of the state space of the target task. We propose A2T (Attend, Adapt and Transfer), an attentive deep architecture which adapts and transfers from these source tasks. Our model is generic enough to effect transfer of either policies or value functions. Empirical evaluations on different learning algorithms show that A2T is an effective architecture for transfer by being able to avoid negative transfer while transferring selectively from multiple source tasks in the same domain.
Transferring knowledge from prior source tasks in solving a new target task can be useful in several learning applications. The application of transfer poses two serious challenges which have not been adequately addressed. First, the agent should be able to avoid negative transfer, which happens when the transfer hampers or slows down the learning instead of helping it. Second, the agent should be able to selectively transfer, which is the ability to select and transfer from different and multiple source tasks for different parts of the state space of the target task. We propose A2T (Attend, Adapt and Transfer), an attentive deep architecture which adapts and transfers from these source tasks. Our model is generic enough to effect transfer of either policies or value functions. Empirical evaluations on different learning algorithms show that A2T is an effective architecture for transfer by being able to avoid negative transfer while transferring selectively from multiple source tasks in the same domain.
1 INTRODUCTION
One of the goals of Artificial Intelligence (AI) is to build autonomous agents that can learn and adapt to new environments. Reinforcement Learning (RL) is a key technique for achieving such adaptability. The goal of RL algorithms is to learn an optimal policy for choosing actions that maximize some notion of long term performance. Transferring knowledge gained from tasks solved earlier to solve a new target task can help, either in terms of speeding up the learning process or in terms of achieving a better solution, among other performance measures. When applied to RL, transfer could be accomplished in many ways (see Taylor & Stone (2009; 2011) for a very good survey of the field). One could use the value function from the source task as an initial estimate in the target task to cut down exploration [Sorg & Singh (2009)]. Alternatively one could use policies from the source task(s) in the target task. This can take one of two forms - (i) the derived policies can be used as initial exploratory trajectories [Atkeson & Schaal (1997); Niekum et al. (2013)] in the target task and (ii) the derived policy could be used to define macro-actions which may then be used by the agent in solving the target task [Mannor et al. (2004); Brunskill & Li (2014)].
∗Authors contributed equally
While transfer in RL has been much explored, there are two crucial issues that have not been adequately addressed in the literature. The first is negative transfer, which occurs when the transfer results in a performance that is worse when compared to learning from scratch in the target task. This severely limits the applicability of many transfer techniques only to cases for which some measure of relatedness between source and target tasks can be guaranteed beforehand. This brings us to the second problem with transfer, which is the issue of identifying an appropriate source task from which to transfer. In some scenarios, different source tasks might be relevant and useful for different parts of the state space of the target task. As a real world analogy, consider multiple players (experts) who are good at different aspects of a game (say, tennis). For example, Player 1 is good at playing backhand shots while Player 2 is good at playing forehand shots. Consider the case of a new player (agent) who wants to learn tennis by selectively learning from these two experts. We handle such a situation in our architecture by allowing the agent to learn how to pick and use solutions from multiple and different source tasks while solving a target task, selectively applicable for different parts of the state space. We call this selective transfer. Our agent can transfer knowledge from Player 1 when required to play backhand shots and Player 2 for playing forehand shots. Further, let us consider consider the situation that both Player 1 and Player 2 are bad at playing drop shots. Apart from the source tasks, we maintain a base network that learns from scratch on the target task. The agent can pick and use the solution of the base network when solving the target task at the parts of the state space where transferring from the source tasks is negative. Such a situation could arise when the source task solutions are irrelevant for solving the target task over a specific portion of the state space, or when the transferring from the source tasks is negative over a specific portion of the state space (for example, transferring the bad drop shot abilities of Players 1 and 2). This situation also entails the first problem of avoiding negative transfer. Our framework allows an agent to avoid transferring from both Players 1 and 2 while learning to play drop shots, and rather acquire the drop shot skill by learning to use the base network. The architecture is trained such that the base network uses not just the experience obtained through the usage of its solutions in the target task, but the overall experience acquired using the combined knowledge of the source tasks and itself. This enables the base network solutions to get closer to the behavior of the overall architecture (which uses the source task solutions as well). This makes it easier for the base network to assist the architecture to fine tune the useful source task solutions to suit the target task perfectly over time.
The key contribution in the architecture is a deep attention network, that decides which solutions to attend to, for a given input state. The network learns solutions as a function of current state thereby aiding the agent in adopting different solutions for different parts of the state space in the target task.
To this end, we propose A2T: Attend, Adapt and Transfer, an Attentive Deep Architecture for Adaptive Transfer, that avoids negative transfer while performing selective transfer from multiple source tasks in the same domain. In addition to the tennis example, A2T is a fairly generic framework that can be used to selectively transfer different skills available from different experts as appropriate to the situation. For instance, a household robot can appropriately use skills from different experts for different household chores. This would require the skill to transfer manipulation skills across objects, tasks and robotic actuators. With a well developed attention mechanism, the most appropriate and helpful combination of object-skill-controller can be identified for aiding the learning on a related new task. Further, A2T is generic enough to effect transfer of either action policies or actionvalue functions, as the case may be. We also adapt different algorithms in reinforcement learning as appropriate for the different settings and empirically demonstrate that the A2T is effective for transfer learning for each setting.
2 RELATED WORK
As mentioned earlier, transfer learning approaches could deal with transferring policies or value functions. For example, Banerjee & Stone (2007) describe a method for transferring value functions by constructing a Game tree. Similarly, Sorg & Singh (2009) use the value function from a source task as the initial estimate of the value function in the target task.
Another method to achieve transfer is to reuse policies derived in the source task(s) in the target task. Probabilistic Policy Reuse as discussed in Fernández & Veloso (2006) maintains a library of policies and selects a policy based on a similarity metric, or a random policy, or a max-policy from the knowledge obtained. This is different from the proposed approach in that the proposed approach
can transfer policies at the granularity of individual states which is not possible in policy-reuse rendering it unable to learn customized policy at that granularity.Atkeson & Schaal (1997); Niekum et al. (2013) evaluated the idea of having the transferred policy from the source tasks as explorative policies instead of having a random exploration policy. This provides better exploration behavior provided the tasks are similar. Talvitie & Singh (2007) try to find the promising policy from a set of candidate policies that are generated using different action mapping to a single solved task. In contrast, we make use of one or more source tasks to selectively transfer policies at the granularity of state. Apart from policy transfer and value transfer as discussed above, Ferguson & Mahadevan (2006) discuss representation transfer using Proto Value Functions.
The idea of negative and selective transfer have been discussed earlier in the literature. For example, Lazaric & Restelli (2011) address the issue of negative transfer in transferring samples for a related task in a multi-task setting. Konidaris et al. (2012) discuss the idea of exploiting shared common features across related tasks. They learn a shaping function that can be used in later tasks.
The two recent works that are very relevant to the proposed architecture are discussed in Parisotto et al. (2015) and Rusu et al. (2016). Parisotto et al. (2015) explore transfer learning in RL across Atari games by trying to learn a multi-task network over the source tasks available and directly finetune the learned multi-task network on the target task. However, fine-tuning as a transfer paradigm cannot address the issue of negative transfer which they do observe in many of their experiments. Rusu et al. (2016) try to address the negative transfer issue by proposing a sequential learning mechanism where the filters of the network being learned for an ongoing task are dependent through lateral connections on the lower level filters of the networks learned already for the previous tasks. The idea is to ensure that dependencies that characterize similarity across tasks could be learned through these lateral connections. Even though they do observe better transfer results than direct fine-tuning, they are still not able to avoid negative transfer in some of their experiments.
3 PROPOSED ARCHITECTURE
Let there be N source tasks and let K1,K2, . . .KN be the solutions of these source tasks 1, . . . N respectively. Let KT be the solution that we learn in the target task T . Source tasks refer to tasks that we have already learnt to perform and target task refers to the task that we are interested in learning now. These solutions could be for example policies or state-action values. Here the source tasks should be in the same domain as the target task, having the same state and action spaces. We propose a setting where KT is learned as a function of K1, . . . ,KN ,KB , where KB is the solution of a base network which starts learning from scratch while acting on the target task. In this work, we use a convex combination of the solutions to obtain KT .
KT (s) = wN+1,sKB(s) + N∑ i=1 wi,sKi(s) (1)
N+1∑ i=1 wi,s = 1, wi,s ∈ [0, 1] (2)
wi,s is the weight given to the ith solution at state s.
The agent uses KT to act in the target task. Figure 1a shows the proposed architecture. While the source task solutionsK1, . . . ,KN remain fixed, the base network solutions are learnt and henceKB can change over time. There is a central network which learns the weights (wi,s, i ∈ 1, 2, . . . , N+1), given the input state s. We refer to this network as the attention network. The [0, 1] weights determine the attention each solution gets allowing the agent to selectively accept or reject the different solutions, depending on the input state. We adopt a soft-attention mechanism whereby more than one weight can be non-zero [Bahdanau et al. (2014)] as opposed to a hard-attention mechanism [Mnih et al. (2014)] where we are forced to have only one non-zero weight.
wi,s = exp (ei,s) N+1∑ j=1 exp (ej,s) , i ∈ {1, 2, . . . , N + 1} (3)
(e1,s, e2,s, . . . , eN+1,s) = f(s; θa) (4)
Here, f(s; θa) is a deep neural network (attention network), which could consist of convolution layers and fully connected layers depending on the representation of input. It is parametrised by θa and takes as input a state s and outputs a vector of length N + 1, which gives the attention scores for the N + 1 solutions at state s. Eq.(3) normalises this score to get the weights that follow Eq.(2).
If the ith source task solution is useful at state s, then wi,s is set to a high value by the attention network. Working at the granularity of states allows the attention network to attend to different source tasks, for different parts of the state space of the target task, thus giving it the ability to perform selective transfer. For parts of the state space in the target task, where the source task solutions cause negative transfer or where the source task solutions are not relevant, the attention network learns to give high weight to the base network solution (which can be learnt and improved), thus avoiding negative transfer.
Depending on the feedback obtained from the environment upon following KT , the attention network’s parameters θa are updated to improve performance.
As mentioned earlier, the source task solutions, K1, . . . ,KN remain fixed. Updating these source task’s parameters would cause a significant amount of unlearning in the source tasks solutions and result in a weaker transfer, which we observed empirically. This also enables the use of source task solutions, as long as we have the outputs alone, irrespective of how and where they come from.
Even though the agent follows KT , we update the parameters of the base network that produces KB , as if the action taken by the agent was based only on KB . Due to this special way of updating KB , apart from the experience got through the unique and individual contribution of KB to KT in parts of the state space where the source task solutions are not relevant, KB also uses the valuable experience got by using KT which uses the solutions of the source tasks as well.
This also means that, if there is a source task whose solution Kj is useful for the target task in some parts of its state space, then KB tries to replicate Kj in those parts of the state space. In practise, the source task solutions though useful, might need to be modified to suit perfectly for the target task. The base network takes care of these modifications required to make the useful source task solutions perfect for the target task. The special way of training the base network assists the architecture in achieving this faster. Note that the agent could follow/useKj throughKT even when KB does not attain its replication in the corresponding parts of the state space. This allows for a good performance of the agent in earlier stages training itself, when a useful source task is available and identified.
Since the attention is soft, our model has the flexibility to combine multiple solutions. The use of deep neural networks allow the model to work even for large, complex RL problems. The deep attention network, allows the agent to learn complex selection functions, without worrying about
representation issues a priori. To summarise, for a given state, A2T learns to attend to specific solutions and adapts this attention over different states, hence attaining useful transfer. A2T is general and can be used for transfer of solutions such as policy and value.
3.1 POLICY TRANSFER
The solutions that we transfer here are the source task policies, taking advantage of which, we learn a policy for the target task. Thus, we have K1, . . . ,KN ,KB ,KT ← π1, . . . πN , πB , πT . Here π represents a stochastic policy, a probability distribution over all the actions. The agent acts in the target task, by sampling actions from the probability distribution πT . The target task policy πT is got as described in Eq.(1) and Eq.(2). The attention network that produces the weights for the different solutions, is trained by the feedback got after taking action following πT . The base network that produces πB is trained as if the sampled action came from πB (though it originally came from πT ), the implications of which were discussed in the previous section. When the attention network’s weight for the policy πB is high, the mixture policy πT is dominated by πB , and the base network learning is nearly on-policy. In the other cases, πB undergoes off-policy learning. But if we look closely, even in the latter case, since πB moves towards πT , it tries to be nearly on-policy all the time. Empirically, we observe that πB converges. This architecture for policy transfer can be used alongside any algorithm that has an explicit representation of the policy. Here we describe two instantiations of A2T for policy transfer, one for direct policy search using REINFORCE algorithm and another in the Actor-Critic setup.
3.1.1 POLICY TRANSFER IN REINFORCE ALGORITHMS USING A2T:
REINFORCE algorithms [Williams (1992)] can be used for direct policy search by making weight adjustments in a direction that lies along the gradient of the expected reinforcement. The full architecture is same as the one shown in Fig.1a with K ← π. We do direct policy search, and the parameters are updated using REINFORCE. Let the attention network be parametrized by θa and the base network which outputs πB be parametrized by θb. The updates are given by:
θa ← θa + αθa(r − b) ∂ ∑M t=1 log(πT (st, at))
∂θa (5)
θb ← θb + αθb(r − b) ∂ ∑M t=1 log(πB(st, at))
∂θb (6)
where αθa , αθb are non-negative factors, r is the return obtained in the episode, b is some baseline and M is the length of the episode. at is the action sampled by the agent at state st following πT . Note that while πT (st, at) is used in the update of the attention network, πB(st, at) is used in the update of the base network.
3.1.2 POLICY TRANSFER IN ACTOR-CRITIC USING A2T:
Actor-Critic methods [Konda & Tsitsiklis (2000)] are Temporal Difference (TD) methods that have two separate components, viz., an actor and a critic. The actor proposes a policy whereas the critic estimates the value function to critique the actor’s policy. The updates to the actor happens through TD-error which is the one step estimation error that helps in reinforcing an agent’s behaviour.
We use A2T for the actor part of the Actor-Critic. The architecture is shown in Fig.1b. The actor, A2T is aware of all the previous learnt tasks and tries to use those solution policies for its benefit. The critic evaluates the action selection from πT on the basis of the performance on the target task. With the same notations as REINFORCE for st, at, θa, θb, αθa , αθb , πB , πT ; let action at dictated by πT lead the agent to next state st+1 with a reward rt+1 and let V (st) represent the value of state st and γ the discount factor. Then, the update equations for the actor are as below:
δt = rt+1 + γV (st+1)− V (st) (7)
θa ← θa + αθaδt ∂ log πT (st,at) ∂θa∣∣∣∂ log πT (st,at)∂θa ∣∣∣ (8)
θb ← θb + αθbδt ∂ log πB(st,at) ∂θb∣∣∣∂ log πB(st,at)∂θb ∣∣∣ (9) Here, δt is the TD error. The state-value function V of the critic is learnt using TD learning.
3.2 VALUE TRANSFER
In this case, the solutions being transferred are the source tasks’ action-value functions, which we will call as Q functions. Thus, K1, . . . ,KN ,KB ,KT ← Q1, . . . , QN , QB , QT . Let A represent the discrete action space for the tasks and Qi(s) = {Q(s, aj) ∀ aj ∈ A}. The agent acts by using QT in the target task, which is got as described in Eq.(1) and Eq.(2). The attention network and the base network of A2T are updated as described in the architecture.
3.2.1 VALUE TRANSFER IN Q LEARNING USING A2T:
The state-action value Q function is used to guide the agent to selecting the optimal action a at a state s, whereQ(s, a) is a measure of the long-term return obtained by taking action a at state s. One way to learn optimal policies for an agent is to estimate the optimal Q(s, a) for the task. Q-learning [Watkins & Dayan (1992)] is an off-policy Temporal Difference (TD) learning algorithm that does so. The Q-values are updated iteratively through the Bellman optimality equation [Puterman (1994)] with the rewards obtained from the task as below:
Q(s, a)← E[r(s, a, s′) + γmaxa′Q(s′, a′)]
In high dimensional state spaces, it is infeasible to update Q-value for all possible state-action pairs. One way to address this issue is by approximating Q(s, a) through a parametrized function approximator Q(s, a; θ),thereby generalizing over states and actions by operating on higher level features [Sutton & Barto (1998)]. The DQN [Mnih et al. (2015)] approximates the Q-value function with a deep neural network to be able to predict Q(s, a) over all actions a, for all states s.
The loss function used for learning a Deep Q Network is as below: L(θ) = Es,a,r,s′ [ ( yDQN −Q(s, a; θ) )2 ],
with yDQN = ( r + γmaxa′Q(s′, a′, θ−) ) Here, L represents the expected TD error corresponding to current parameter estimate θ. θ− represents the parameters of a separate target network, while θ represents the parameters of the online network. The usage of a target network is to improve the stability of the learning updates. The gradient descent step is shown below:
∇θL(θ) = Es,a,r,s′ [(yDQN −Q(s, a; θ))∇θQ(s, a)]
To avoid correlated updates from learning on the same transitions that the current network simulates, an experience replay [Lin (1993)] D (of fixed maximum capacity) is used, where the experiences are pooled in a FIFO fashion.
We use DQN to learn our expertsQi, i ∈ 1, 2 . . . N on the source tasks. Q-learning is used to ensure QT (s) is driven to a good estimate of Q functions for the target task. Taking advantage of the offpolicy nature of Q-learning, both QB and QT can be learned from the experiences gathered by an -greedy behavioral policy based on QT . Let the attention network that outputs w be parametrised by θa and the base network outputting QB be parametrised by θb. Let θa− and θb− represent the parameters of the respective target networks. Note that the usage of target here is to signify the parameters (θ−a , θ − b ) used to calculate the target value in the Q-learning update and is different from its usage in the context of the target task. The update equations are:
yQT = (r + γmaxa′QT (s′, a′; θa−, θb−)) (10)
LQT (θa, θb) = Es,a,r,s′ [(yQT −QT (s, a; θa, θb))2] (11)
LQB (θb) = Es,a,r,s′ [(yQT −QB(s, a; θb))2] (12)
∇θaLQT = E[(yQT −QT (s, a))∇θaQT (s, a)] (13)
∇θbLQB = E[(yQT −QB(s, a))∇θbQR(s, a)] (14) θa and θb are updated with the above gradients using RMSProp. Note that the Q-learning updates for both the attention network (Eq.(11)) and the base network (Eq.(12)) use the target value generated by QT . We use target networks for both QB and QT to stabilize the updates and reduce the nonstationarity as in DQN training. The parameters of the target networks are periodically updated to that of the online networks.
4 EXPERIMENTS AND DISCUSSION
We evaluate the performance of our architecture A2T on policy transfer using two simulated worlds, viz., chain world and puddle world as described below. The main goal of these experiments is to test the consistency of results with the algorithm motivation. Chain world: Figure 2a shows the chain world where the goal of the agent is to go from one point in the chain (starting state) to another point (goal state) in the least number of steps. At each state the agent can choose to either move one position to the left or to the right. After reaching the goal state the agent gets a reward that is inversely proportional to the number of steps taken to reach the goal.
Puddle worlds: Figures 2b and 2c show the discrete version of the standard puddle world that is widely used in Reinforcement Learning literature. In this world, the goal of the agent is to go from a specified start position to the goal position, maximising its return. At each state the agent can choose one of these four actions: move one position to the north, south, east or west.With 0.9 probability the agent moves in the chosen direction and with 0.1 probability it moves in a random direction irrespective of its choice of action. On reaching the goal state, the agent gets a reward of +10. On reaching other parts of the grid the agent gets different penalties as mentioned in the legend of the figures. . We evaluate the performance of our architecture on value transfer using the Arcade Learning Environment (ALE) platform [Bellemare et al. (2012)]. Atari 2600: ALE provides a simulator for Atari 2600 games. This is one of the most commonly used benchmark tasks for deep reinforcement learning algorithms [Mnih et al. (2015), Mnih et al. (2016), Parisotto et al. (2015), Rusu et al. (2016)]. We perform our adaptive transfer learning experiments on the Atari 2600 game Pong.
4.1 ABILITY TO DO SELECTIVE TRANSFER
In this section, we consider the case when multiple partially favorable source tasks are available such that each of them can assist the learning process for different parts of the state space of the target task. The objective here is to first show the effectiveness of the attention network in learning to focus only on the source task relevant to the state the agent encounters while trying to complete the target task and then evaluating the full architecture with an additional randomly initialised base network.
This is illustrated for the Policy Transfer setting using the chain world shown in (Fig. 2a). Consider that the target task LT is to start in A or B with uniform probability and reach C in the least number of steps. Now, consider that two learned source tasks, viz., L1 and L2, are available. L1 is the source task where the agent has learned to reach the left end (A) starting from the right end (B). In contrast, L2 is the source task where the agent has learned to reach the right end (B) starting from the left end (A). Intuitively, it is clear that the target task should benefit from the policies learnt for tasks L1 and L2. We learn to solve the task LT using REINFORCE given the policies learned for L1 and L2. Figure 3a (i) shows the weights given by the attention network to the two source task policies for different parts of the state space at the end of learning. We observe that the attention network has learned to ignore L1, and L2 for the left, and right half of the state space of the target task, respectively. Next, we add base network and evaluate the full architecture on this task. Figure 3a (ii) shows the weights given by the attention network to the different source policies for different parts of the state space at the end of learning. We observe that the attention network has learned to ignore L1, and L2 for the left, and right half of the state space of the target task, respectively. As the base network replicates πT over time, it has a high weight throughout the state space of the target task.
We also evaluate our architecture in a relatively more complex puddle world shown in Figure 2c. In this case, L1 is the task of moving from S1 to G1, and L2 is the task of moving from S2 to G1. In the target task LT , the agent has to learn to move to G1 starting from either S1 or S2 chosen with uniform probability. We learn the task LT using Actor-Critic method, where the following are available (i) learned policy for L1 (ii) learned policy for L2 and (iii) a randomly initialized policy network (the base network). Figure 3b shows the performance results. We observe that actor-critic using A2T is able to use the policies learned for L1, and L2 and performs better than a network learning from scratch without any knowledge of source tasks.
We do a similar evaluation of the attention network, followed by our full architecture for value transfer as well. We create partially useful source tasks through a modification of the Atari 2600 game Pong. We take inspiration from a real world scenario in the sport Tennis, where one could imagine two different right-handed (or left) players with the first being an expert player on the forehand but weak on the backhand, while the second is an expert player on the backhand but weak on the forehand. For someone who is learning to play tennis with the same style (right/left) as the experts, it is easy to follow the forehand expert player whenever he receives a ball on the forehand and follow the backhand expert whenever he receives a ball on the backhand.
We try to simulate this scenario in Pong. The trick is to blur the part of the screen where we want to force the agent to be weak at returning the ball. The blurring we use is to just black out all pixels in the specific region required. To make sure the blurring doesn’t contrast with the background, we modify Pong to be played with a black background (pixel value 0) instead of the existing gray (pixel value 87). We construct two partially helpful source task experts L1 and L2. L1 is constructed by
training a DQN on Pong with the upper quadrant (the agent’s side) blurred, while L2 is constructed by training a DQN with the lower quadrant (the agent’s side) blurred. This essentially results in the ball being invisible when it is in the upper quadrant for L1 and lower quadrant for L2. We therefore expect L1 to be useful in guiding to return balls on the lower quadrant, and L2 for the upper quadrant. The goal of the attention network is to learn suitable filters and parameters so that it will focus on the correct source task for a specific situation in the game. The source task experts L1 and L2 scored an average of 9.2 and 8 respectively on Pong game play with black background. With an attention network to suitably weigh the value functions of L1 and L2, an average performance of 17.2 was recorded just after a single epoch (250,000 frames) of training. (The score in Pong is in the range of [−21, 21]). This clearly shows that the attention mechanism has learned to take advantage of the experts adaptively. Fig. 4 shows a visualisation of the attention weights for the same.
We then evaluate our full architecture (A2T) in this setting, i.e with an addition of DQN learning from scratch (base network) to the above setting. The architecture can take advantage of the knowledge of the source task experts selectively early on during the training while using the expertise of the base network wherever required, to perform well on the target task. Figure 5 summarizes the results, where it is clear that learning with both the partially useful experts is better than learning with only one of them which in turn is better than learning from scratch without any additional knowledge.
4.2 ABILITY TO AVOID NEGATIVE TRANSFER AND ABILITY
TO TRANSFER FROM FAVORABLE TASK
We first consider the case when only one learned source task is available such that its solution K1 (policy or value) can hamper the learning process of the new target task. We refer to such a source task as an unfavorable source task. In such a scenario, the attention network shown in Figure 1a should learn to assign a very low weight (ignore) to K1 . We also consider a modification of this setting by adding another source task whose solution K2 is favorable to the target task. In such a scenario, the attention network should learn to assign high weight (attend) to K2 while ignoringK1.
We now define an experiment using the puddle world from Figure 2b for policy transfer. The target task in our experiment is to maximize the return in reaching the goal state G1 starting from any one of the states S1, S2, S3, S4. We artificially construct an unfavorable source task by first learning to solve the above task and then negating the weights of the topmost layer of the actor network. We then add a favorable task to the above setting. We artificially construct a favorable source task
simply by learning to solve the target task and using the learned actor network. Figure 6 shows the results. The target task for the value transfer experiment is to reach expert level performance on Pong. We construct two kinds of unfavorable source tasks for this experiment. Inverse-Pong: A DQN on Pong trained with negated reward functions, that is with R′(s, a) = −R(s, a) where R(s, a) is the reward provided by the ALE emulator for choosing action a at state s. Freeway: An expert DQN on another Atari 2600 game, Freeway, which has the same range of optimal value functions and same action space as Pong. We empirically verified that the Freeway expert DQN leads to negative transfer when directly initialized and fine-tuned on Pong which makes this a good proxy for a negative source task expert even though the target task Pong has a different state space.
We artificially construct a favorable source task by learning a DQN to achieve expertise on the target task (Pong) and use the learned network. Figure 7a compares the performance of the various scenarios when the unfavorable source task is Inverse-Pong, while Figure 7b offers a similar comparison with the negative expert being Freeway.
From all the above results, we can clearly see that A2T does not get hampered by the unfavorable source task by learning to ignore the same and performs competitively with just a randomly initialized learning on the target task without any expert available. Secondly, in the presence of an additional source task that is favorable, A2T learns to transfer useful knowledge from the same while ignoring the unfavorable task, thereby reaching expertise on the target task much faster than the other scenarios.
4.3 VISUALIZATION: EVOLUTION OF ATTENTION WEIGHTS WITH ONE POSITIVE AND ONE NEGATIVE EXPERT
We present the evolution of attention weights for the experiment described in Section 4.2 where we focus on the efficacy of the A2T framework in providing an agent the ability to avoid negative transfer and transfer from a favorable source task (perfect expert). Figure 8 depicts the evolution of
the attention weights (normalised in the range of [0, 1]) during the training of the A2T framework. The corresponding experiment is the case where the target task is to solve Pong, while there are two source task experts, one being a perfect Pong playing trained DQN (to serve as positive expert), and the other being the Inverse-Pong DQN trained with negated reward functions (to serve as negative expert). Additionally, there’s also the base network that learns from scratch using the experience gathered by the attentively combined behavioral policy from the expert networks, the base network and itself.
We train the framework for 30 epochs, and the plot illustrates the attention weights every second epoch. We clearly see from figure 8 that there is no weird co-adaptation that happens in the training, and the attention on the negative expert is uniformly low throughout. Initially, the framework needs to collect some level of experience to figure out that the positive expert is optimal (or close to optimal). Till then, the attention is mostly on the base network, which is learning from scratch. The attention then shifts to the positive expert which in turn provides more rewarding episodes and transition tuples to learn from. Finally, the attention drifts slowly to the base network from the positive expert again, after which the attention is roughly random in choosing between the execution of positive expert and the base network. This is because the base network has acquired sufficient expertise as the positive expert which happens to be optimal for the tar-
get task. This visualization clearly shows that A2T is a powerful framework in ignoring a negative expert throughout and using a positive expert appropriately to learn quickly from the experience gathered and acquire sufficient expertise on the target task.
4.4 WHEN A PERFECT EXPERT IS NOT AVAILABLE AMONG THE SOURCE TASKS
In our experiments in the previous subsection dealing with prevention of negative transfer and using a favorable source task, we consider the positive expert as a perfect (close to optimal) expert on the same task we treat as the target task. This raises the question of relying on the presence of a perfect expert as a positive expert. If we have such a situation, the obvious solution is to execute each of the experts on the target task and vote for them with probabilities proportional to the average performance of each.
The A2T framework is however generic and not intended to just do source task selection. We illustrate this with an additional baseline experiment, where the positive source task is an imperfect expert on the target task. In such a case, just having a weighted average voting among the available source task networks based on their individual average rewards is upper bounded by the
performance of the best available positive expert, which happens to be an imperfect expert on the target task. Rather, the base network has to acquire new skills not present in the source task networks. We choose a partially trained network on Pong, that scores an average of 8 (max: 21). The graph in figure 9 clearly shows that the A2T framework with a partial Pong expert and a negative expert performs better than i) learning from scratch, ii) A2T with only one negative expert, and performs worse than A2T with one perfect positive expert and one negative expert. This is expected because
a partial expert cannot provide as much of expert knowledge as a perfect expert, but still provides some useful knowledge in speeding the process of solving the target task. An important conclusion from this experiment is that the A2T framework is capable of discovering new skills not available among any of the experts when such skills are required for optimally solving the target task. To maintain consistency, we perform the same number of runs for averaging scores and experimented with both learning rates and pick the better performing one (0.00025).
5 CONCLUSION AND FUTURE WORK
In this paper we present a very general deep neural network architecture, A2T, for transfer learning that avoids negative transfer while enabling selective transfer from multiple source tasks in the same domain. We show simple ways of using A2T for policy transfer and value transfer. We empirically evaluate its performance with different algorithms, using simulated worlds and games, and show that it indeed achieves its stated goals. Apart from transferring task solutions, A2T can also be used for transferring other useful knowledge such as the model of the world.
While in this work we focused on transfer between tasks that share the same state and action spaces and are in the same domain, the use of deep networks opens up the possibility of going beyond this setting. For example, a deep neural network can be used to learn common representations [Parisotto et al. (2015)] for multiple tasks thereby enabling transfer between related tasks that could possibly have different state-action spaces. A hierarchical attention over the lower level filters across source task networks while learning the filters for the target task network is another natural extension to transfer across tasks with different state-action spaces. The setup from Progressive Neural Networks [Rusu et al. (2016)] could be borrowed for the filter transfer, while the A2T setup can be retained for the policy/value transfer. Exploring this setting for continuous control tasks so as to transfer from modular controllers as well avoid negative transfer is also a potential direction for future research.
The nature of tasks considered in our experiments is naturally connected to Hierarchical Reinforcement Learning and Continual Learning. For instance, the blurring experiments inspired from Tennis based on experts for specific skills like Forehand and Backhand could be considered as learning from sub-goals (program modules) like Forehand and Backhand to solve a more complex and broader task like Tennis by invoking the relevant sub-goals (program modules). This structure could be very useful to build a household robot for general purpose navigation and manipulation whereby specific skills such as manipulation of different objects, navigating across different source-destination points, etc could be invoked when necessary. The attention network in the A2T framework is essentially a soft meta-controller and hence presents itself as a powerful differentiable tool for Continual and Meta Learning. Meta-Controllers have typically been been designed with discrete decision structure over high level subgoals. This paper presents an alternate differentiable meta-controller with a soft-attention scheme. We believe this aspect can be exploited for differentiable meta-learning architectures for hierarchical reinforcement learning. Over all, we believe that A2T is a novel way to approach different problems like Transfer Learning, Meta-Learning and Hierarchical Reinforcement Learning and further refinements on top of this design can be a good direction to explore.
ACKNOWLEDGEMENTS
Thanks to the anonymous reviewers of ICLR 2017 who have provided thoughtful remarks and helped us revise the paper. We would also like to thank Sherjil Ozair, John Schulman, Yoshua Bengio, Sarath Chandar, Caglar Gulchere and Charu Chauhan for useful feedback about the work.
APPENDIX A: DETAILS OF THE NETWORK ARCHITECTURE IN VALUE TRANSFER EXPERIMENTS
For the source task expert DQNs, we use the same architecture as [Mnih et al. (2015)] where the input is 84 × 84 × 4 with 32 convolution filters, dimensions 8 × 8, stride 4 × 4 followed by 64 convolution filters with dimensions 4× 4 and stride 2× 2, again followed by 64 convolution filters of size 3×3 and stride 1×1. This is then followed by a fully connected layer of 512 units and finally by a fully connected output layer with as many units as the number of actions in Pong (Freeway) which is 3. We use ReLU nonlinearity in all the hidden layers.
With respect to the A2T framework architecture, we have experimented with two possible architectures:
• The base and attention networks following the NIPS architecture of Mnih et al. (2013) except that the output layer is softmax for the attention network.
• The base and attention networks following the Nature architecture of Mnih et al. (2015) with a softmax output layer for the attention network.
Specifically, the NIPS architecture of Mnih et al. (2013) takes in a batch of 84 × 84 × 4 inputs, followed by 16 convolution filters of dimensions 8× 8 with stride 4× 4, 32 convolution filters with dimensions 4 × 4 and stride 2 × 2, a fully connected hidden layer of 256 units, followed by the output layer. For the Selective Transfer with Blurring experiments described in Section 4.1, we use the second option above. For the other experiments in Section 4.2 and the additional experiments in Appendix, we use the first option. The attention network has N + 1 outputs where N is the number of source tasks.
APPENDIX B: TRAINING DETAILS
TRAINING ALGORITHM
For all our experiments in Value Transfer, we used RMSProp as in [Mnih et al. (2015)] for updating gradient. For Policy Transfer, since the tasks were simple, stochastic gradient descent was sufficient to provide stable updates. We also use reward clipping, target networks and experience replay for our value transfer experiments in exactly the same way (all hyper parameters retained) as [Mnih et al. (2015)]. A training epoch is 250,000 frames and for each training epoch, we evaluate the networks with a testing epoch that lasts 125,000 frames. We report the average score over the completed episodes for each testing epoch. The average scores obtained this way are averaged over 2 runs with different random seeds. In the testing epochs, we use = 0.05 in the -greedy policy.
LEARNING RATE
In all our experiments, we trained the architecture using the learning rates, 0.0025 and 0.0005. In general, the lower learning rate provided more stable (less variance) training curves. While comparing across algorithms, we picked the best performing learning rate out of the two (0.0025 and 0.0005) for each training curve.
APPENDIX C: BLURRING EXPERIMENTS ON PONG
The experts are trained with blurring (hiding the ball) and black background as illustrated in APPENDIX A. Therefore, to compare the learning with that of a random network without any additional knowledge, we ran the baseline DQN on Pong with a black background too. Having a black background provides a rich contrast between the white ball and the black background, thereby making training easier and faster, which is why the performance curves in that setting are different to the other two settings reported for Inverse Pong and Freeway Negative transfer experiments where no blacking is done and Pong is played with a gray background. The blurring mechanism in Pong is illustrated in APPENDIX E.
APPENDIX E: BLURRING MECHANISM IN PONG - DETAILS
APPENDIX D: BLURRING EXPERIMENTS ON BREAKOUT
Similar to our Blurring experiment on Pong, we additionally ran another experiment on the Atari 2600 game, Breakout, to validate the efficiency of our attention mechanism. We consider a setup with two experts L1 and L2 along with our attention network. The experts L1 and L2 were trained by blurring the lower left and right quadrants of the breakout screen respectively. We don’t have to make the background black like in the case of Pong because the background is already black in Breakout and direct blurring is sufficient to hiding the ball in the respective regions without any contrasts introduced. We blur only the lower part so as to make it easy for the agent to at least anticipate the ball based on the movement at the top. We empirically observed that blurring the top half (as well) makes it hard to learn any meaningful partially useful experts L1 and L2.
The goal of this experiment is to show that the attention network can learn suitable filters so as to dynamically adapt and learn to select the expert appropriate to the situation (game screen) in the task. The expert L1 which was blurred on the left bottom half is bound to weak at returning balls on that region while L2 is expected to be weak on the right. This is in the same vein as the forehandbackhand example in Tennis and its synthetic simulation for Pong by blurring the upper and lower quadrants. During game play, the attention mechanism is expected to ignore L2 when the ball is on the bottom right half (while focusing on L1) and similarly ignore L2 (while focusing on L1) when the ball is on the left bottom half. We learn experts L1 and L2 which score 42.2 and 39.8 respectively. Using the attention mechanism to select the correct expert, we were able to achieve a score of 94.5 after training for 5 epochs. Each training epoch corresponds to 250, 000 decision steps, while the scores are averaged over completed episodes run for 125, 000 decision steps. This shows that the attention mechanism learns to select the suitable expert. Though the performance is limited by the weaknesses of the respective experts, our goal is to show that the attention paradigm is able to take advantage of both experts appropriately. This is evident from the scores achieved by standalone experts and the attention mechanism. Additionally, we also present a visualization of the attention mechanism weights assigned to the experts L1 and L2 during game play in APPENDIX G. The weights assigned are in agreement with what we expect in terms of selective attention. The blurring mechanism is visually illustrated in APPENDIX F.
APPENDIX G: BLURRING ATTENTION VISUALIZATION ON BREAKOUT
APPENDIX F: BLURRING MECHANISM IN BREAKOUT - DETAILS
APPENDIX J: CASE STUDY OF TARGET TASK PERFORMANCE LIMITED BY DATA AVAILABILITY

In this paper a well known soft mixture of experts model is adapted for, and applied to, a specific type of transfer learning problem in reinforcement learning (RL), namely transfer of action policies and value functions between similar tasks. Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably.
One possible implication of this work is that architecture and even learning algorithm choices could simply be stated in terms of the objective of the target task, rather than being hand-engineered by the experimenter. This is clearly an interesting direction of future work which the paper illuminates.
Pros:
The paper diligently explains how the network architecture fits in with various widely used reinforcement learning setups, which does facilitate continuation of this work.
The experiments are good proofs of concept, but do not go beyond that i.m.h.o.
Even so, this work provides convincing clues that collections of deep networks, which were trained on not entirely different tasks, generalize better to related tasks when used together rather than through conventional transfer learning (e.g. fine-tuning).
Cons:
As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks. Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell.
The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016).
Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). It would be more illuminating to consider tasks where final performance is plausibly limited by data availability. It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task.
Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data.