POLICY TRANSFER WITH STRATEGY OPTIMIZATION
Computer simulation provides an automatic and safe way for training robotic control policies to achieve complex tasks such as locomotion. However, a policy trained in simulation usually does not transfer directly to the real hardware due to the differences between the two environments. Transfer learning using domain randomization is a promising approach, but it usually assumes that the target environment is close to the distribution of the training environments, thus relying heavily on accurate system identification. In this paper, we present a different approach that leverages domain randomization for transferring control policies to unknown environments. The key idea that, instead of learning a single policy in the simulation, we simultaneously learn a family of policies that exhibit different behaviors. When tested in the target environment, we directly search for the best policy in the family based on the task performance, without the need to identify the dynamic parameters. We evaluate our method on five simulated robotic control problems with different discrepancies in the training and testing environment and demonstrate that our method can overcome larger modeling errors compared to training a robust policy or an adaptive policy.
The authors propose a policy transfer scheme which in the source domain simultaneously learns a family of policies parameterised by dynamics parameters and then employs an optimisation framework to select appropriate dynamics parameters based on samples from the target domain. The approach is evaluated on a number of simulated transfer tasks (either transferring from DART to MuJoCo or by introducing deliberate model inaccuracies).
This is interesting work in the context of system identification for policy transfer with an elaborate experimental evaluation. The policy learning part seems largely similar to that employed by Yu et al. 2017 (as acknowledged by the authors). This makes the principal contribution, in the eyes of this reviewer, the optimisation step conducted based on rollouts in the target domain. While the notion of optimising over the space of dynamics parameters is intuitive the question arises whether this optimisation step makes for a substantive contribution over the original work. This point is not really addressed in the experimental evaluation as benchmarking is performed against a robust and an adaptive policy but not explicitly against the (arguably) most closely related work in Yu et al. It could be argued, of course, that Yu et al. essentially use adaptive policy generation but they do explicitly learn dynamics parameters based on recent history of actions and observations. An explicit comparison therefore seems appropriate (or alternatively a discussion of why it is not required).
Another point which would, in my view, add significant value is explicit discussion of the baseline performances observed in the various experiments. For example, in the hopper experiment (Sec 5.2) the authors state that the baseline methods were not able to adapt to the new environment. Real value could be derived here if the authors could elaborate on why this is the case. The same applies in Sec 5.3-5.6. 
(I would add here, as an aside, that I thought the notion in Sec 5.6 of framing the learning of policies for handling deformable objects as a transfer task based on rigid objects to be a nice idea. And not one this reviewer has come across before - though this could merely be a reflection of limited familiarity with the literature).
The experimental evaluation seems thorough with the above caveat of a seemingly missing benchmark in Yu et al. I would also encourage the authors to add more detail in the experimental section in the main text specifically with regards to number of trials run to arrive at variances in the figures as well as what metric these shaded areas actually signify. 
A minor point: the J in equ 1 seems (to me at least) undefined. I suspect that it signifies the expected cumulative reward and was meant to be introduced in Sec 3 where the J may have been dropped from the latex?
If the above points were addressed I think this would make a valuable and interesting contribution to the ICLR community. As it stands I believe it is marginally below the acceptance threshold.
[ADDENDUM: given the author feedback and addition of the benchmark experiments requested I have updated my score.]
Pros:
———
- interesting work
- accessible
- effective
- thorough evaluation (though potentially missing a key benchmark)
Cons:
———
- potentially missing a key benchmark (and therefore seems somewhat incremental)
- only limited insight offered by the authors in the discussion of the experimental results
- some more details needed with regards to the experimental setup