LEARNING TO SIMULATE
Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications.
Pros:
* Using RL to choose the simulator parameters is a good idea. It does not sound too novel, but at the same time I am not personally aware of this having been explored in the past (Note that my confidence is 4, so maybe other reviewers might be able to chime in on this point)
* In theory, you don't need domain adaptation or other sim2real techniques if you manage to get the optimal parameters of the simulator with this method.
* Certain attributes of the method were evaluated sufficiently: eg the number of training epochs for each policy iteration, the dataset size generated in each iteration, and whether initialization was random or not in each iteration.
Cons:
* Experiments were underwhelming, and the choice of problems/parameters to tune was not the right one for the problem.
* Parts of the paper could be clearer
QUALITY:
* I believe that although the idea is great, but the quality of the experiments could have been higher. Firstly, better problems could have been selected to showcase the method. I was excited to see experiments with CARLA, but was underwhelmed when I realized that the only parameter of the simulator that the method controlled was the number and the type of cars in the scene, and the task of interest was a car counting task (for which not much detail was provided). This would have been much more interesting and useful to the community if more parameters, including rendering parameters (like lighting, shading, textures, etc) were part of the search space. Similarly, the semantic segmentation task could have used more than one category. But even for the one category, there were no previous methods considered, and the only comparison was between random parameters and the learned ones, where we only see marginal improvement, and what I perceive to be particularly low IoU for the car (although it'd help to know what's the SOTA there for comparison) For both vision applications I could help but wonder why the authors did not try to simply train on the  validation set to give us another datapoint to evaluate the performance of the method: this is data that *is* used for training the outer loop, so it does beg the question of what is the advantage of having hte inner loop. 
CLARITY:
* The writing of the paper was clear for the most part, however the experimental section could have been clearer. I was wondering how model/hyperparameter selection was performed? Was there another validation set (other than the one used to train the outer loop)
* The proposed policy is dubbed "its". What does it mean?
* It's not clear what is a "deliberately adversarial" initialization. Could you elaborate?
* The letter R is used to mean "reward" and "rendering". This is confusing. Similarly some symbols are not explicitly explained (eg S) Generally Section 2.3 is particularly unclear and confusing until one gets to the experimental section.
* Section 3 discusses the technique and states that "we can thus generate or oversample unusual situations that would otherwise not be part of the training data" I believe it is important to state that, as the method is presented, this is only true if the "validation" data is varied enough and includes such situations. I believe this would be more applicable if eg rendering parameters were varied and matched the optimal ones.
* Also the method is presented as orthogonal to domain adaptation and other sim-to-real techniques. However, I do not necessarily believe that this paper should be discussed outside the context of such techniques like domain randomization, Cycada, PixelDA etc. Even though these (esp. the latter ones) focus on vision, I do think it sets the right context.
ORIGINALITY:
* As far as I'm aware noone has tried something similar yet. However, I'm not confident on this.
SIGNIFICANCE:
* Although the idea is good, I don't think that the approach to select the simulation parameters presented in the experiments in such a way is significant. I think that eg doing so for rendering parameters would be a lot more powerful and useful (and probably a lot more challenging). Also, I think that a single set of parameters (which seems to be what the goal is in this work) is not what one wants to achieve; rather one wants to find a good range of parameters that can help in the downstream task.