Conventional reinforcement learning rarely considers how the physical variations in the environment (eg. mass, drag, etc.) affect the policy learned by the agent. In this paper, we explore how changes in the environment affect policy generalization. We observe experimentally that, for each task we considered, there exists an optimal environment setting that results in the most robust policy that generalizes well to future environments. We propose a novel method to exploit this observation to develop robust actor policies, by automatically developing a sampling curriculum over environment settings to use in training. Ours is a model-free approach and experiments demonstrate that the performance of our method is on par with the best policies found by an exhaustive grid search, while bearing a significantly lower computational cost.
The paper looks at the problem of generalization across physical parameter varaition in learning for continuous control. The paper presents a method to develop a sampling based curriculum over env. settings for training robust agents. 
* The paper makes an interesting observation on inadvertent generalization in robust policy learning. 
However, the examples in both the cartpole and the pendulum cases seem not to be watertight. 
For instance, the authors claim that 
But from a dynamical system perspective in both cases, the controller is operating near limits. 
The solution and subsequent generalization depend more on the topology of the solution space. 
A heavy Pendulum is an overdamped system and required the policy to operate at the limits of action to generate momentum for swing up. Hence a solution for a lighter pendulum in implicitly included. Similarly, the rolling ball is an underdamped system, and where the policy operates near zero limits in light ball case to prevent the system from going unstable. Adding mass results in damping which makes it easier. In this case, as well the solution space is implicitly contained.
But this is not a novel observation. Similar observations have been made for Robust control and Model-Reference Adaptive Control. 
The paper also overlooks a number of related works in model-free randomization [4], adaptive randomization [3], adversarial randomization [5,6]. The method also does not compare with model-based methods for adaptive policy learning and iLQR based methods to handle this problem [2, 7].
The argument that the method is model-free is perhaps not as acceptable since the model parameters need to be known apriori for adaptation. The policy itself may be model-free but that is a design choice. 
A good experimental evaluation for this is generalization across known unknowns and unknown unknowns. 
* The algorithm itself is reasonable but the problem setup and choice of a discrete dynamics parameter choices are questionable. The bandit style method operates over a discrete decision set. 
It also assumes in the multi-parameter setting that they are independent, which may not be true very often. 
The algorithm proposed itself isnt novel, but would have been justified if the results supported the use of such a method. 
* Experiments are quite weak. 
Both the experimental domains are rather simplistic with smooth nonlinear dynamics. There are more sophisticated and interesting continuous control environments such as control suite [1] or manipulation suite [2].  
It would be useful to see how tis method works in more complicated domains and how the performance compares with simpler methods such as joint brute-force randomization both in performance and in computation.  
Questions: 
1. Please provide details of Algorithm 1. How are the quantities K and M related? 
2. What is the process of task initialization? What information is required and what priors are used. Uniform prior over what range?
In summary, the authors explore an interesting adaptive curriculum design method. However, in its current form, the work needs more thought and empirical evaluation for the sake of completeness. 
References:
1. Model Reference Adaptive Control [https://doi.org/10.1007/978-1-4471-5102-9_116-1
]
2. ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems [https://arxiv.org/abs/1707.04674]
3. EPOpt: Learning Robust Neural Network Policies Using Model Ensembles [https://arxiv.org/abs/1610.01283]
4. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
[https://arxiv.org/abs/1610.01283]
5. Certifying Some Distributional Robustness with Principled Adversarial Training [https://arxiv.org/pdf/1710.10571.pdf]
6. Adversarially Robust Policy Learning: Active Construction of Physically-Plausible Perturbations [http://vision.stanford.edu/pdf/mandlekar2017iros.pdf]
7. Synthesis and Stabilization of Complex Behaviors through Online Trajectory Optimization [https://homes.cs.washington.edu/~todorov/papers/TassaIROS12.pdf]