Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters . It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of trajectories which reflect the differences in various possible environments, and then selecting some subset of these to learn robust policies, such as the ones that result in the worst performance. We propose an active learning based framework, EffAcTS, to selectively choose model parameters for this purpose so as to collect only as much data as necessary to select such a subset. We apply this framework to an existing method, namely EPOpt, and experimentally validate the gains in sample efficiency and the performance of our approach on standard continuous control tasks. We also present a Multi-Task Learning perspective to the problem of Robust Policy Search, and draw connections from our proposed framework to existing work on Multi-Task Learning.
This paper targets at a particular type of robust policy search, where a simulation environment exists with explicit tuning parameters, which is referred to as the model parameters of a Markov decision process. The task of robust policy search is to learn a policy robust to all the parameters of the simulator, so that it can potentially give robust performance in real environment. The previous work handles this problem by sampling many trajectories and only learning from the trajectories, in which the current policy produces the worst performance. This approach effectively focus the policy search on the worst case performance, but is highly inefficient as most of the sampled trajectories are discarded. This paper proposes to improve the sampling efficiency by building a surrogate model predicting the return of the current policy given a MDP parameter. The surrogate model is used to select the MDP parameters leading to the worsts performance, so that the policy search can directly sample and learn from the selected MDP parameters without discarding any trajectories.
This paper tries to tackle the sampling efficiency of RL with building a probabilistic surrogate model. This is a promising direction. The biggest concern is that this paper tackles this problem with a combination with existing techniques, leaving many questions unanswered. Presenting the paper in a more theoretical-grounded perspective would make the paper much stronger.
This paper uses a linear stochastic bandits (LSB) method to build a surrogate model of the return of the current policy and fits the surrogate model into the EPOpt framework by sampling from the worst performing parameters according to the surrogate model. As the Thompson sampling algorithm of LSB draw samples from the distribution of MDP parameters that leads to the wost performance, why not directly use it for policy search?
The surrogate model is expected not to give accurate prediction everywhere due to the limited number of data but produces uncertainty of its prediction as an dictator. However, the uncertainty of prediction is not used by the proposed algorithm.
The presentation of the experiment section needs to be improved. The performance of the baseline needs to be explicitly presented, otherwise it is hard to compare. The proposed method will not outperform if the number of trajectories used for updating policy is the same, as the surrogate model can never be as good as the real model. It would be nice to explicitly demonstrate the runtime and performance trade-off.
Minor issues:
1. What does TRPO stand for?
2. When referring to the paper instead of the authors, the citation format needs to be (authors year) instead of authors (year).