Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings. But does the distribution over environment settings contain important biases, and do these lead to agents that fail in certain cases despite high average-case performance? In this work, we consider worst-case analysis of agents over environment settings in order to detect whether there are directions in which agents may have failed to generalize. Specifically, we consider a 3D first-person task where agents must navigate procedurally generated mazes, and where reinforcement learning agents have recently achieved human-level average-case performance. By optimizing over the structure of mazes, we find that agents can suffer from catastrophic failures, failing to find the goal even on surprisingly simple mazes, despite their impressive average-case performance. Additionally, we find that these failures transfer between different agents and even significantly different architectures. We believe our findings highlight an important role for worst-case analysis in identifying whether there are directions in which agents have failed to generalize. Our hope is that the ability to automatically identify failures of generalization will facilitate development of more general and robust agents. To this end, we report initial results on enriching training with settings causing failure.
Update:
I appreciate the clarifications and the extension of the paper in response to the reviews. I think it made the work stronger. The results in the newly added section are interesting and actually suggest that by putting more effort into training set design/augmentation, one could further robustify the agents, possibly up to the point where they do not break at all in unnatural ways. It is a pity the authors have not pushed the work to this point (and therefore the paper is not as great as it could be), but still I think it is a good paper that can be published.
-----
The paper analyzes the performance of modern reinforcement-learning-based navigation agents by searching for “adversarial” maze layouts in which the agents do not perform well. It turns out that such mazes  exist, and moreover, one can find even relatively simple maze configurations that are easily solved by humans, but very challenging for the algorithms.
Pros:
1) Interesting and relevant topic: it is important not only to push for best results on benchmarks, but also understand the limitations of existing approaches.
2) The paper is well written
3) The experiments are quite thorough and convincing. I especially appreciate that it is demonstrated that there exist simple mazes that can be easily solved by humans, but not by algorithms. The analysis of transferability of “adversarial” mazes between different agents is also a plus.
Cons:
1) I am not convinced worst-case performance is the most informative way to evaluate models. Almost no machine learning model is perfect, and therefore almost for any model it would be possible to find training or validation samples on which it does not perform well. Why is it so surprising that this is also the case for navigation models? Why would one assume they should be perfect? Especially given that the generated “adversarial” mazed lie outside of the training data distribution, seemingly quite far outside. Are machine learning models ever expected to perfectly generalize outside of the training data distribution? Very roughly speaking, the key finding of the paper can be summarized as “several recent navigation agents have problems finding and entering small rooms of the type they never saw during training” - is this all that significant?
To me, the most interesting part of the paper is that the models generalize as well as they do. I would therefore like to see if it is possible to modify the training distribution - by adding “adversarial” mazes, potentially in an iterative fashion, or just by hand-designing a wider distribution of mazes - so that generalization becomes nearly perfect and the proposed search method is not anymore able to find “adversarial” mazes that are difficult for the algorithm, but easy for humans.
2) On a related note, to me the largest difference between the mazes generated in this paper and the classical adversarial images is that the modification of the maze is not constrained to be small or imperceptible. In fact, it is quite huge - the generated mazes are far from the training distribution. This is a completely different regime. This is like training a model to classify cartoon images and then asking it to generalize to real images (or perhaps other way round). Noone would expect existing image classification models to do this. This major difference with classical adversarial examples should be clearly acknowledged.
3) It would be interesting to know the computational requirements of the search method. I guess it can be estimated from the information in the paper, but would be great to mention it explicitly. (I am sorry if it is already mentioned and I missed it)
To conclude, I think the paper is interesting and well executed, but the presented results are very much to be expected. To me the most interesting aspect of the work is that the navigation agents generalize surprisingly well. Therefore, I believe the work would be much more useful if it focused more on how to make the agents generalize even better, especially since there is a very straightforward way to try this - by extending the training set. I am currently in the borderline mode, but would be very happy to change my evaluation if the focus of the paper is somewhat changed and additional experiments on improving generalization (or some other experiments, but making the results a bit more useful/surprising) are added.