Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries
Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.
 This paper studies the problem of robust policy optimization, motivated by the fact that policies that work in simulations do not transfer well to real world. The authors propose to use the diversity measure on the roll out trajectories to select a diverse set of simulator configurations and train policies that can work well for all of those selected configurations. Overall, I think the idea is interesting but it is not entirely clear why adding diverse configurations should result in good performance, and the experiments are very limited and not convincing enough. Pros: - The paper is easy to follow. - The idea of using a diverse summary to do robust policy optimization is interesting. - The diversity measure on the trajectories instead of the space of configuration parameters also intuitively makes sense since it takes into account that the similarity between two configuration parameters does not typically mean the similarity between their corresponding policies. Cons: - The setting of this paper seems to only work for the fully observable case with state space being in R^d, deterministic dynamics and deterministic policy (otherwise the diversity measure would be stochastic?). It would be good to clarify these in Sec. 2. - For the example in Fig 2 and the first experiment, what I don*t understand is why the initial state is not part of the policy. - It is not clear if the reason that EP-OPT performed worse than the proposed approach is only because there are not enough rollouts for EP-OPT. This could be an unfair comparison. - It would be good to show the comparison to EP-OPT for the second experiment as well. - Two experiments may not be enough to verify valid performance since there could be a lot of randomness in the results. - In page 6, it would be good to clarify that the summary being optimal is only with respect to f(M_s), but not the original problem of finding optimal policy.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries
Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper studies the problem of robust policy optimization, motivated by the fact that policies that work in simulations do not transfer well to real world. The authors propose to use the diversity measure on the roll out trajectories to select a diverse set of simulator configurations and train policies that can work well for all of those selected configurations. Overall, I think the idea is interesting but it is not entirely clear why adding diverse configurations should result in good performance, and the experiments are very limited and not convincing enough. Pros: - The paper is easy to follow. - The idea of using a diverse summary to do robust policy optimization is interesting. - The diversity measure on the trajectories instead of the space of configuration parameters also intuitively makes sense since it takes into account that the similarity between two configuration parameters does not typically mean the similarity between their corresponding policies. Cons: - The setting of this paper seems to only work for the fully observable case with state space being in R^d, deterministic dynamics and deterministic policy (otherwise the diversity measure would be stochastic?). It would be good to clarify these in Sec. 2. - For the example in Fig 2 and the first experiment, what I don*t understand is why the initial state is not part of the policy. - It is not clear if the reason that EP-OPT performed worse than the proposed approach is only because there are not enough rollouts for EP-OPT. This could be an unfair comparison. - It would be good to show the comparison to EP-OPT for the second experiment as well. - Two experiments may not be enough to verify valid performance since there could be a lot of randomness in the results. - In page 6, it would be good to clarify that the summary being optimal is only with respect to f(M_s), but not the original problem of finding optimal policy.
assistant
 This paper studies the problem of robust policy optimization, motivated by the fact that policies that work in simulations do not transfer well to real world. The authors propose to use the diversity measure on the roll out trajectories to select a diverse set of simulator configurations and train policies that can work well for all of those selected configurations. Overall, I think the idea is interesting but it is not entirely clear why adding diverse configurations should result in good performance, and the experiments are very limited and not convincing enough. Pros: - The paper is easy to follow. - The idea of using a diverse summary to do robust policy optimization is interesting. - The diversity measure on the trajectories instead of the space of configuration parameters also intuitively makes sense since it takes into account that the similarity between two configuration parameters does not typically mean the similarity between their corresponding policies. Cons: - The setting of this paper seems to only work for the fully observable case with state space being in R^d, deterministic dynamics and deterministic policy (otherwise the diversity measure would be stochastic?). It would be good to clarify these in Sec. 2. - For the example in Fig 2 and the first experiment, what I don*t understand is why the initial state is not part of the policy. - It is not clear if the reason that EP-OPT performed worse than the proposed approach is only because there are not enough rollouts for EP-OPT. This could be an unfair comparison. - It would be good to show the comparison to EP-OPT for the second experiment as well. - Two experiments may not be enough to verify valid performance since there could be a lot of randomness in the results. - In page 6, it would be good to clarify that the summary being optimal is only with respect to f(M_s), but not the original problem of finding optimal policy.