Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries
Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.
 This paper addresses the problem of finding a policy that will perform well in a real environment when training in a simulator that may have errors. It takes the now standard approach of trying to find a policy that performs well in an ensemble of simulated environments that are perturbations of the basic simulator. The question is: how can we construct an ensemble that represents the uncertainty about the real world well while being small enough for computational efficiency. The idea is to construct a diverse set of samples that represents the whole space of important variations in the simulation; the particular novelty here is to ensure that the sample set attains coverage over the *behaviors* of the simulator rather than the parameters of the simulation. This problem is made difficult by the fact that there is no finite space of samples to choose from and the fact that we don*t have a natural distance metric on the simulator behavior. The main positive contributions of the paper are: - The view of the problem of selecting from an infinite set as one of streaming sub-modular optimization. This is a nice idea that is new to me and seemed appropriate for the problem. - The idea that we want diversity in behavior, and then the technical approach of defining a kernel on simulator parameter sets that depends on the trajectories that those parameters induce. I do have a set of questions and concerns: - Might it not be better (more robust) to use not just trajectories from the current policy, but from other policies as well, to compute the kernel on parameter sets? - How do you get the length-scale parameters for the kernel? - The confidence intervals in table 1 are too big to really justify firm conclusions; it would be better to run the algorithms several more times, until the intervals pull apart. - You say: "For ease of implementation and since, in higher dimensional system, the variance of the policy gradients becomes a significant factor, we train the robot on both the environment summaries and the N_s random rollouts." This seems like it might be an important point that should be addressed earlier. And, why does this ease implementation? - I didn*t understand: "Diverse summaries are more consistent than pure random sampling." What do you mean by consistent here? - The metrics used in the empirical comparisons don*t seem exactly right to me. The goal of this work is to learn a policy that is robust in some sense (so that, e.g., it will do sim-2-real well). We really want it to work well in all possible cases, not just in expectation or according to the sampling distribution you create, (I guess---since the paper said the minimax criterion was desirable but difficult to work with). So, then, it seems like the best performance criterion would be to sample a whole lot of domain parameters and report performance on the worst (rather than reporting performance on a distribution that*s like the one you trained it on or on an easy random one). Overall, my view is that the idea is good, but somewhat small, and it hasn*t really yet been proven to make a big difference.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries
Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper addresses the problem of finding a policy that will perform well in a real environment when training in a simulator that may have errors. It takes the now standard approach of trying to find a policy that performs well in an ensemble of simulated environments that are perturbations of the basic simulator. The question is: how can we construct an ensemble that represents the uncertainty about the real world well while being small enough for computational efficiency. The idea is to construct a diverse set of samples that represents the whole space of important variations in the simulation; the particular novelty here is to ensure that the sample set attains coverage over the *behaviors* of the simulator rather than the parameters of the simulation. This problem is made difficult by the fact that there is no finite space of samples to choose from and the fact that we don*t have a natural distance metric on the simulator behavior. The main positive contributions of the paper are: - The view of the problem of selecting from an infinite set as one of streaming sub-modular optimization. This is a nice idea that is new to me and seemed appropriate for the problem. - The idea that we want diversity in behavior, and then the technical approach of defining a kernel on simulator parameter sets that depends on the trajectories that those parameters induce. I do have a set of questions and concerns: - Might it not be better (more robust) to use not just trajectories from the current policy, but from other policies as well, to compute the kernel on parameter sets? - How do you get the length-scale parameters for the kernel? - The confidence intervals in table 1 are too big to really justify firm conclusions; it would be better to run the algorithms several more times, until the intervals pull apart. - You say: "For ease of implementation and since, in higher dimensional system, the variance of the policy gradients becomes a significant factor, we train the robot on both the environment summaries and the N_s random rollouts." This seems like it might be an important point that should be addressed earlier. And, why does this ease implementation? - I didn*t understand: "Diverse summaries are more consistent than pure random sampling." What do you mean by consistent here? - The metrics used in the empirical comparisons don*t seem exactly right to me. The goal of this work is to learn a policy that is robust in some sense (so that, e.g., it will do sim-2-real well). We really want it to work well in all possible cases, not just in expectation or according to the sampling distribution you create, (I guess---since the paper said the minimax criterion was desirable but difficult to work with). So, then, it seems like the best performance criterion would be to sample a whole lot of domain parameters and report performance on the worst (rather than reporting performance on a distribution that*s like the one you trained it on or on an easy random one). Overall, my view is that the idea is good, but somewhat small, and it hasn*t really yet been proven to make a big difference.
assistant
 This paper addresses the problem of finding a policy that will perform well in a real environment when training in a simulator that may have errors. It takes the now standard approach of trying to find a policy that performs well in an ensemble of simulated environments that are perturbations of the basic simulator. The question is: how can we construct an ensemble that represents the uncertainty about the real world well while being small enough for computational efficiency. The idea is to construct a diverse set of samples that represents the whole space of important variations in the simulation; the particular novelty here is to ensure that the sample set attains coverage over the *behaviors* of the simulator rather than the parameters of the simulation. This problem is made difficult by the fact that there is no finite space of samples to choose from and the fact that we don*t have a natural distance metric on the simulator behavior. The main positive contributions of the paper are: - The view of the problem of selecting from an infinite set as one of streaming sub-modular optimization. This is a nice idea that is new to me and seemed appropriate for the problem. - The idea that we want diversity in behavior, and then the technical approach of defining a kernel on simulator parameter sets that depends on the trajectories that those parameters induce. I do have a set of questions and concerns: - Might it not be better (more robust) to use not just trajectories from the current policy, but from other policies as well, to compute the kernel on parameter sets? - How do you get the length-scale parameters for the kernel? - The confidence intervals in table 1 are too big to really justify firm conclusions; it would be better to run the algorithms several more times, until the intervals pull apart. - You say: "For ease of implementation and since, in higher dimensional system, the variance of the policy gradients becomes a significant factor, we train the robot on both the environment summaries and the N_s random rollouts." This seems like it might be an important point that should be addressed earlier. And, why does this ease implementation? - I didn*t understand: "Diverse summaries are more consistent than pure random sampling." What do you mean by consistent here? - The metrics used in the empirical comparisons don*t seem exactly right to me. The goal of this work is to learn a policy that is robust in some sense (so that, e.g., it will do sim-2-real well). We really want it to work well in all possible cases, not just in expectation or according to the sampling distribution you create, (I guess---since the paper said the minimax criterion was desirable but difficult to work with). So, then, it seems like the best performance criterion would be to sample a whole lot of domain parameters and report performance on the worst (rather than reporting performance on a distribution that*s like the one you trained it on or on an easy random one). Overall, my view is that the idea is good, but somewhat small, and it hasn*t really yet been proven to make a big difference.