Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing nearoptimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.
Summary
This paper addresses the problem of transfer in RL. After an agent is given an opportunity to train from a distribution of environments, we want an agent to perform well on the test environment. This paper specifically focuses on the setting where the state space, action space, reward space, and discount factor are the same across all environments, while the transition dynamics may differ. An environment's transition dynamics is assumed to depend on a hidden parameter that is not observed by the agent, in contrast to some previous work which assumes observability.
The idea of the main algorithm is as follows. During the training phase, a series of environments is presented. The agent is aware of the demarcations between the environments, but not of the identity (i.e., hidden parameter) of the environments themselves. A "probing" policy is run for a specified number of time steps. From the trajectory generated, a VAE is used to estimate the hidden parameter governing the transition dynamics. For the remaining time in the environment instance, a master policy, conditioned on the estimated parameter, is trained on the reward (any usual RL algorithm suffices; DDQN is used in the experiments).
The novelties of the approach are the problem setting (no access to optimal policies, immediate deployment at test-time), the joint training of a probing policy and VAE model, and a universal policy conditioned on the estimated hidden parameters. The experiments were conducted on domains that were, individually, both continuous and discontinuous with respect to the hidden parameters. The paper concludes that the proposed method improves upon baselines in terms of speed of reward attainment and computation time. Ablations were run for some of the components of the proposed algorithm. 
 
Decision: Weak Reject
 
The two key reasons for the Weak Reject were the following. 
1. Statistically insignificant results (3 runs)
2. Questions about baselines used (see question 6 below)
The proposed algorithm is promising and attempts to directly address limitations in previous literature. The paper thoroughly contextualizes and motivates the current approach in light of previous work in the literature and discusses possible advantages: simplicity and generality; the ability to deploy immediately at test-time instead of having to train another policy (from learning a universal policy); assuming that the hidden parameters are unobservable, which is more realistic; not assuming access to optimal value functions/policies; the algorithm is model-free, so one can avoid the computational and memory overhead (when compared to BNN) of learning a model and avoid compounding model error. I also appreciated the thorough discussion of the key algorithmic choices and their trade-offs in section 3. The presentation of ideas is clear throughout.
 
However, weak experiments keep this paper at a weak reject. The paper claims that the proposed algorithm "significantly outperforms" the baselines (abstract); however, more runs are needed to substantiate these claims. Only 3 runs were made for the test episode, which is not enough to be able to make empirical claims with non-trivial confidence (see https://arxiv.org/abs/1709.06560).  As well, since baseline methods are adapted from the original papers, more thorough hyperparameter sweeps should be performed. I wasn’t sure what “coarse coordinate search” was (pg. 6), but in general it would be good to write down the hyperparameters swept over. 
Some additional questions about the experiments:
1. Did you try oracle with lower dimensional hidden parameter embedding? (understanding more why oracle does possibly worse in HIV and Acrobot)
2. There is the claim on page 2 that the proposed algorithm is "orders of magnitude faster than best model-based method". What is the best model-free method?
3. Is the universal policy of SEPT optimized during test time?
4. Did you manage to replicate the results of DPT in Yao (2018)? Were architectures the same?
5. What is the significance of not requiring rewards at test-time?
6. Why were BNN, MAML, and EPOpt the baselines chosen? What about the other works mentioned in the related work section, like Tirinzoni (2018) or Paul (2019)? Did you try using an LSTM as a baseline to directly learn the policy? One could imagine treating the existence of a hidden parameter as inducing a partial observability problem. 
7. What is a "coarse coordinate search" over hyperparameters? What hyperparameters were tried?
8. Did you try regular DQN? DDQN w/o prioritized replay? 
9. Why were the cumulative reward curves not as significant for Acrobot and HIV as compared to 2D navigation?
10. Why different probe length settings for 2D navigation vs. Acrobot and HIV? How did these hyperparameters affect the time to solve?
11. Why do 2D-room and Acrobot have terminal rewards of 1000 and 10?
I would be willing to raise the score if the key experimental problems I noted were addressed. 
 
Minor comments/questions that did not impact the score
1. Should include the table of computational time in the main body if possible since computational efficiency is one of the core claims
2. Move DDQN comment in 3.2 to the experiments section
3. How does the probing policy approach relate to work in active perception? (https://link.springer.com/article/10.1007/s10514-017-9666-5)
EDIT: After the comments below which addressed my concerns, I have raised the score for this paper from a weak reject to an accept.