Efficient and robust policy transfer remains a key challenge in reinforcement learning. Policy transfer through warm initialization, imitation, or interacting over a large set of agents with randomized instances, have been commonly applied to solve a variety of Reinforcement Learning (RL) tasks. However, this is far from how behavior transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here we seek to answer the question: Will learning to combine adaptation reward with environmental reward lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can "Adapt-to-Learn", that is adapt the source policy to learn to solve a target task with significant transition differences and uncertainties. We show through theory and experiments that our method leads to a significantly reduced sample complexity of transferring the policies between the tasks.
Summary
-------
The authors propose an algorithm for transferring policies across domains differing in their transition model. The idea is to "regularize" the target policy being learned to generate a trajectory distribution similar to the one induced by the source optimal policy in the respective MDP. The method is proved effective in standard simulated robotic tasks (Mujoco).
Major comments
--------------
The problem of transfer in RL is fundamental for scaling RL algorithms to real domains. The presented ideas are interesting and could potentially inspire other works in this field. The paper is well-written and easy to read. However, I believe there might be some technical issues with the derivations that overall make me doubt about the significance of this work. My detailed comments follow.
1. Under Assumption 1, the step from (6) to (7) seems flawed. The definition of the KL divergence requires that p is absolutely continuous w.r.t. q. However, under Ass. 1, whenever the trajectory \tau contains an action that is not taken by \pi^*, q(\tau) = 0, which implies that p(\tau) must be zero as well. The only way to enforce this is to set \pi_\theta = \pi^*, and the solution is trivial. The main issue is that (6) writes the log term of the KL using different action random variables for the numerator and the denominator (a and a'), but these must be the same. Therefore, one cannot simply set to 1 the optimal source actions. Notice also that the action space is defined continuous, hence the value of the pdf in the chosen deterministic action is infinity, not 1.
2. (12) seems to be wrong too. Note that the reward function defined here (r') contains the policy, hence it is a function of \theta and must be differentiated. So the standard REINFORCE estimator written here does not work theoretically.
3. Assumption 2 basically requires that either we can get new samples from the source domain "on-demand", or we have a good simulated model. Both assumptions are often violated in practice. Could the authors elaborate more on why they believe these assumptions to be reasonable?
4. Related to the previous point, assuming the one-step transition models to be Gaussian is convenient, and often reasonable, in practice. But how do we choose the std \sigma? What if the true models are deterministic?
5. I would disagree with the last sentence of section 3.2.1, which states that minimizing the KL divergence is more sample efficient than standard RL/policy-search methods. One can always define an MDP where the immediate reward is the KL term and that is a standard RL problem. Furthermore, if we compute the expected return when the reward is (8), decomposing the log into the sum of two logs, one of the two expectations we obtain is of the form: E_\pi[-(s_{t+1} -s'_{t+1})^2], which is a standard "tracking" problem.
6. I was quite confused by the theoretical section. The introduction states that the proposed method enjoys a sample complexity of O(nH), where I guess n is the number of state-action pairs. Where does that dependence appear in Section 4? Theorem 4.1 is a standard supervised learning bound. How do we read that in RL? What is C? Is |\Pi| the cardinality of the set of policies (assumed finite)? Or is it the pseudo-dimension of the policy space? Lemma 4.3 is also a know result in the literature, but what does it tell us about the proposed approach? The idea is that if I have an accurate model \hat{p} I am also close in value function to the true MDP. But what is \hat{p} here? The proposed algorithm is not computing any explicit transition model.
Minor comments
--------------
- In the last sentence of the introduction, it is not clear what n is in O(nH)
- In the preliminaries, is the reward R: S x A -> R_+ assumed to be positive or is that a typo?
- MDPs are defined with a discount factor \gamma which is however missing in (1) and the next equation (V)
- In the preliminaries, it is not clear whether the rewards differ or not between domains. It is important to specify that they do not.
- In (2), T is defined to map R^n -> R^n. What is n? Is it the number of discrete states? Wasn't the state space continuos?
- I noticed that many sentences in the paper begin without a space after the period of the previous sentence.
- In section 3, the KL divergence (e.g., eq. 3,7) is written as an expectation that contains the pdf p(\tau). Either only the log term is written in the expectation, or E is replaced by an integral.
- In (12), the sum should be up to H and not infinity