We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator. We place a variety of obstacles in the environment for the prey to hide behind, and we only give the agents partial observations of their environment using an egocentric perspective. Although we train the model to play this game from scratch, experiments and visualizations suggest that the agent learns to predict its own visibility in the environment. Furthermore, we quantitatively analyze how agent weaknesses, such as slower speed, effect the learned policy. Our results suggest that, although agent weaknesses make the learning problem more challenging, they also cause more useful features to be learned.
# Review ICLR20, Visual Hide and Seek
This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.
## Overall
**Summary**
The authors introduce a new RL environment and task, "Visual Hide and Seek", in which they analyze how the agent's learned visual representations are impacted by its speed, auxiliary rewards, and opponent behavior.
**Overall Opinion**
This paper presents a thorough analysis and great visualizations of agent behaviors and representations under different conditions. I wish more papers would put this much effort into analyzing their agents. I'd highly recommend this paper get accepted since I believe the analysis carried out here and the conclusions reached are quite novel and the paper is overall well-written.
However, at the same time, the work of [Baker et al., 2019][1] was published with significantly more fanfare. I hope their work does not overshadow this one since they are only related in the general task concept.
[1]: https://arxiv.org/abs/1909.07528
Some major issues I had with this work:
- In general, please run more random seeds. Just reporting on a single random seed is not enough, as per [Henderson et al., 2018][2].
- There are some sections of the paper where the order of paragraphs is confusing. You start the introduction by stating what you've done and letting the reader wonder "why?". The explanation is only given in the second paragraph. So I'd suggest rotating the second paragraph upwards before the first. Similarly, at the beginning of section 4, you just mention the results - this should either be shorter (1 sentence, as an overview of the work in this section) or moved to the end of that section.
- You're missing a section about future work and flaws/problems of your work at the very end (the latter if which should be in "Discussion"), which is common to include in ICLR publications.
[2]: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16669/16677
Here are some minor...
## Specific comments and questions
### Abstract
all good
### Intro
- Fig.1: What is that white thing that the seeker/hider have on the capsule?
### Rel. Work
all good
### Method
- What's the speed (FPS) of that Unity engine? Why didn't you use Mujuco/(Py)Bullet/Gym-Miniworld?
- You need to add some measurements and units: the arena size doesn't have a unit, the size (diameter) of the hiders/seekers is unclear, turning left/right is unclear (how far left/right, after action-repeat)
- You mention any real-valued position to be valid - so the agents can step into obstacles? And how about through obstacles?
- "Affordance Learning" is usually not used for static environment geometry like obstacles (e.g. [Georgia-Tech course on "Human-Robot Interaction"][3])
- Why 4-layer CNNs? Why not 6 or 8 or a ResNet? Would you think the features would be stronger/weaker in an 8-layer CNN?
[3]: https://www.cc.gatech.edu/~athomaz/classes/CS8803-HRI-Spr08/MayaChandan/Site/Affordance_Learning.html
### Experiments
- 4.1 "... learned this play game" -> "... learned this game".
- 4.2 "mid-level features" -> what's that? The activation of the convolutional kernels after the second layer CNN? What's the dimension? And why did you pick the 2nd layer, not any of the other 3?
- Tab.3: This is averaged over how many frames of rollout?
- "... case can moves a lot faster." -> "... case can move a lot faster".
- Fig.2 is very interesting. Well done.
- Fig.3: the font is not consistent with other figures
- Fig.4: remove the blueish background to increase contrast. Increase the font size of the ticks on the left. Make the legend color boxes slightly bigger. Add more space or a visual divider between the different states - especially on the right side it's hard to make out where one stops and the next begins.
- Fig.4/5: This analysis is lovely and we need more of this in DRL.
- 4.4 in the text, you sometimes write "not S" and sometimes "¬S". Please change the "not s"
- Fig.5: (suggestion) Merge/sum the 2 columns (in both A/B merge the left and right plot into one by summing); subtract the random policy values as you did with Fig.4.
- "We summarize representative cases, and put the full results for all combinations in the Appendix" - no you didn't.
- Fig.6: What is going on in the left third of this diagram? What is this colorful mush? If this is by any chance indicating a change over time, do you maybe want to spread a single, very colorful plot of distance over time into multiple less colorful plots? At least add a legend, please. Also, I'd recommend smoothing (moving average or smoothing spline). 
- Also maybe add reward over time plots, as is common in DRL, to show that your policies converged after 8 mil. steps.
### Conclusion
All good, save for the missing future work and critical analysis of your work.
### Appendix
all good