We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks. We introduce a novel metric between Markov Decision Processes and establish that close MDPs have close optimal value functions, that is that optimal value functions are Lipschitz continuous with respect to tasks. These theoretical results lead us to a value transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate. We illustrate the benefits of the method in Lifelong RL experiments.
Updated review: Thanks to the authors for response. I think it clarifies a few points for me except one, which is that I still dont understand how the ATARI games share the same state space? Under what model is this true? And if the paper really wants to use this as an example, why are there no results on even a single ATARI game. 
------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary: The paper presents a method for lifelong reinforcement learning problems. By lifelong reinforcement learning the paper means that a set of tasks, specified by MDPs, specified by reward and transition function (R and T) are presented to the agent sequentially and the agent must solve them. The idea of the paper is to assume that the tasks that live in the space <R, T> are Lipschitz continuous and if that is so the experience from the previous task can be used to upper bound the Q values for the current task. The paper suggests using the RMax algorithm that can take as input an upper bound (optimistic initialization) on the Q values of the MDP it is trying to solve and solve the MDP efficiently. For the proposed algorithm called LRMax, the paper presents theoretical results on the sample and computational complexity. The experiments on an 11x11 grid world show that LRMax is more efficient compared to other baselines. 
I think this is an interesting paper that takes a principled approach towards a lifelong RL problem. However, I find the paper hard to read and I think that the connection of what paper presents to a real-life problem is missing. 
The assumption of Lipschitz continuity of tasks makes little sense to me. Does this mean that the state and action space of the MDPs that specify these tasks have to be the same always? Is that not a very restrictive set of lifelong RL problems? And if it is tried to remove the restriction just by assuming a very big state and action space and how do we verify the Lipschitz continuity and will the method scale to large state, action spaces?
I also found it difficult to understand if the paper is considering an online setting or an offline setting. It seems to me it is considering an online setting, but then proposition 6 and 7 are called sample and computational complexity. In general, in online setting sample complexity can be expressed in terms of T - time step, or the number of rounds. But the paper presents computational complexity in terms of timesteps. Is it just wrong naming or Is the computational complexity the cost of computation at every time step? But then it includes a T term. For quite a lot of time, I thought that this is computational complexity and the T denotes the transition function until I realized, T in this theorem denotes time step. 
(Using T for transition function and the time step is confusing). 
But then how can computational complexity (that measures the time required) be expressed in terms of time?
The paper claims that it proposes an upper bound on Q values of an MDP that can be computed analytically. I doubt that. First, I think that if this is true then it would have been useful to keep this algorithm in the main body of the paper. I particularly feel that such a bound is a big and important contribution. In any case, it is in the appendix. In the appendix, the paper basically proposes to use dynamic programming to compute the upper bound. So we do assume a model of the transition probabilities of the world that is explicitly maintained and probably learned from the experience of the agent. Does this not restrict the approach of this paper to very small problems for which such tables can be maintained? (This restriction is in addition to the restriction imposed by the assumption of Lipschitz continuity).
Experiments: 
The experiments shows that LRMax is more efficient compared to other baselines when it is presented with sequential MDP to solve. This is nice and expected. I think one of the experiments that would have been interesting is what happens when the tasks are not Lipschitz continuous. I would not expect the algorithm the perform well in such cases, but it would be good to know how poorly does the algorithm performs compared to other baselines. So how robust the proposed algorithm is to the assumption it is specifically designed for?
Minor comments: 
- the paper keeps using the phrase ‘agent explores greedily wrt to Q function.’ I found this confusing, I think the paper meant agent acts greedily wrt to Q function. The world explore seems to indicate that there is a separate exploration phase. 
- The paper claims that distance between two games in arcade learning environment is smaller than the maximum distance between any two MDPs defined on a common state-action space of ALE. Can the paper clarify what this means. Is there a way to verify this claim. What are the common state action space of all games in ALE?
 If it is so, then the LRMax will do great in terms of transferring knowledge in between games on ALE. Why did the paper choose not to show results on ALE then?