Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads to higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification to be a central challenge and motivate it with theoretical analysis. Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without additional exploration on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of interactions. This work demonstrates that Meta RL from observational data is possible and we hope it will gather additional interest from the community to tackle this problem.
This paper studies the meta-RL problem in the off-policy, batch learning setting. Batch-RL is the setting in which a policy is learned entirely offline, that is, without interaction with the environment and given only trajectories collected by some policy. Compared to RL, Meta-RL involves the additional challenge of task-inference; the goal of Meta-RL is to train a policy that can generalize to a distribution of tasks (i.e. a distribution of MDPs), without actually being given a description of the task (unlike contextual multi-task policy learning). A simple approach for solving the meta-RL problem thus might be to first perform task inference by encoding data from some task into a task description, and then condition a contextual policy on this task description. 
The authors state that the straight-forward application of such an idea in the Batch-RL setting fails due to an issue they term the MDP “misidentification” problem, wherein having multiple tasks in a single batch results in confusion between tasks. This issue really only arises in the setting where task inference is jointly learned with the multi-tasking contextual policy. Thus, they propose a stage-wise algorithm wherein first 1) the N tasks in some dataset of trajectories are learned by N separate policies, and then 2) these N separate policies are distilled into a single master policy, wherein the subpolicy enacted by the master policy is modulated by some sort of task description. The distillation procedure thus involves task description (i.e. mapping from task data to a task embedding) and supervised learning (i.e. imitation) of each sub policy when conditioned on respective task embedding.
* Pros
	* Demonstrates the effectiveness of a straightforward stage-wise approach for off-policy meta-RL (albeit without comparison to alternatives)
	* The method seems to perform well on the tasks considered, approaching the level of performance of SOTA model-free algorithms.
	* The approach is general insofar as it could presumably be used with other batch-RL methods (for first stage of algorithm), or even non-batch RL methods. In this sense, the main contribution might be seen as an approach for distilling multiple policies into a single policy in a way that allows for interpolating between them.
	
* Cons
	* Technical novelty: while they address a problem that has not yet seen much attention, their solution is combination existing solutions. I say this because I am not fully convinced that the more novel aspects of their distillation procedure (i.e. the auxiliary task) are absolutely necessary. I would be willing to change my stance on this, given evidence.
		* Ablations of i.e. the auxiliary task would help to clarify this. In all tasks considered, the transition function T does not change. Therefore, the function composite P(E(s_j, a_j)) should actually not learn any task-specific information when predicting s’_j. If the method works without the auxiliary loss, it is very similar to a stagewise version of PEARL, with Batch-RL. 
	* Limited comparisons to relevant alternatives, simple baselines.	 Do not compare to anything other than SAC.
	* Only tasks considered are pointmass, hopper, and half-cheetah; other work (i.e. PEARL) has also been demonstrated on Ant in Mujoco. 
	* Argues that it is more stable than variants that involve interaction with the environment / use critic loss for MDP identification, which is a somewhat unfair since in their case 1) they assume they have data that is good enough to learn a good policy in the batch-RL setting, and 2) the stability is by virtue of the fact that they do things in a stage-wise manner because batch-RL works, i.e. they are doing supervised learning without having to bootstrap.
	* Not sure if the use of term “pre-train” is appropriate, insofar as the test tasks are assumed to come from the same distribution as training tasks. It seems to be more about Batch-Meta-RL. 
	* Not much attention given to attempts to solve the MDP mis-identification problem with simple solutions like giving side-information at training time (i.e. task-ID). 
* Would be helpful to see:
	* More baselines
		* A simple task classifier. This would basically do a nearest neighbor lookup over the training tasks (given a test task), but this might perform well under the reward function.
		* Comparison to something like PEARL, or some method that does involve interaction with the environment -> this would help shed light on whether interaction is necessary for task inference in the the tasks they consider, and if so, such methods would in some sense be oracles.
	* Ablations
		* Remove auxiliary forward prediction loss: if it works without auxiliary loss, this is very similar to a stagewise version of PEARL, with Batch-RL.
		* How much data per task is needed?
	* Harder tasks, where the method can’t approach SAC (as one would expect for sufficiently challenging tasks)
	* Experiment where they study zero-shot generalization by considering disjoint parts of task space
		* Claim zero-shot generalization but in this case they study tasks where resets are not crucial for exploration needed for task inference
		
* Minor comments:
	* I found the discussion about inductive bias in RL at the end of section 3 (last few sentences of last paragraph) to be a bit vague.
I've given a weak reject mainly because 1) auxiliary loss has not been experimentally shown to be crucial and therefore the technical novelty may be relatively limited, and 2) more comparisons are needed, to alternatives or other baselines.  I would be willing to change my decision on this, given supporting evidence.