Producing agents that can generalize to a wide range of visually different environments is a significant challenge in reinforcement learning. One method for overcoming this issue is visual domain randomization, whereby at the start of each training episode some visual aspects of the environment are randomized so that the agent is exposed to many possible variations. However, domain randomization is highly inefficient and may lead to policies with high variance across domains. Instead, we formalize the visual domain randomization problem, and show that minimizing the policy's Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies. We propose a regularization method where the agent is only trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant. We conduct experiments that demonstrate that our technique leads to more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.
This paper proposes a regularization scheme for training vision-based control policies that are robust to variations of the visual input, which the paper classifies as "visual randomization". Visual randomization, as the paper notes, is one of the state-of-the-art techniques proposed from simulation to real robot transfer of vision-based policies trained on synthetic images. The regularization proposed in the paper aims to produce policies that rely on features that are invariant to the visual randomization, so that the resulting behaviour of the agent is consistent across different variations of its visual input for the same states. 
The paper proposes that forcing a policy trained under randomization to have a Lipschitz constant of K, over the randomization parameters, causes the optimal policy to be similar across randomization parameters, with the difference in expected returns of two randomized environments being bounded by K times the maximum difference in their randomization parameters.
I recommend this paper to not be accepted until the following issues are addressed. 
* There are missing details from the experimental setup, which makes the results hard to interpret (see, below). 
* There are  missing details on how vanilla domain randomization was implemented. Domain randomization aims to maximize the expected performance over the environment distribution. This can be implemented properly by computing the expected gradients with data from more than one environment. From the algorithm descriptions in the appendix, it is not clear that this is how vanilla domain randomization was implemented. 
* The title, introduction and conclusions do not reflect the scope of the paper. The paper only addresses situations where the same behaviour is achievable on all environments, an assumption (Mehta et al, 2019) also makes, and its proposed regularization is based on the assumption that the optimal behaviour is achievable with the same policy on all environments. But this is not true in general: for dynamics randomization, different environments may require different policies (e.g. driving a car on a road vs driving off-road). The regularization method may result in overly conservative policies i such situations.  
Questions about experimental details: 
What are the maximum returns  for Cartpole when trained until convergence without randomization? (175? 200? 1000?) If the maximum returns are higher than 175, how does Figure 4 look with more data? This is crucial to understand, for example, the results in Figure 11. That figure shows the proposed regularization slightly hinders the performance for the environments near l=1, g=50 (that region is a darker shade of green on the left subfigure). How do we know if the task has been successfully solved in the green vs purple regions? In all experiments, are the training curves showing the performance of the policies over the same environments (same seeds)? If not, how are the training curves comparable?
Other things to improve:
The conclusions of this paper can be made stronger by adding a comparison with EpOpt-PPO (i.e. optimizing the worst case performance over a set of trajectories sampled from multiple environments)