The ability of modeling the other agents, such as understanding their intentions and skills, is essential to an agent's interactions with other agents. Conventional agent modeling relies on passive observation from demonstrations. In this work, we propose an interactive agent modeling scheme enabled by encouraging an agent to learn to probe. In particular, the probing agent (i.e., a learner) learns to interact with the environment and with a target agent (i.e., a demonstrator) to maximize the change in the observed behaviors of that agent. Through probing, rich behaviors can be observed and are used for enhancing the agent modeling to learn a more accurate mind model of the target agent. Our framework consists of two learning processes: i) imitation learning for an approximated agent model and ii) pure curiosity-driven reinforcement learning for an efficient probing policy to discover new behaviors that otherwise can not be observed. We have validated our approach in four different tasks. The experimental results suggest that the agent model learned by our approach i) generalizes better in novel scenarios than the ones learned by passive observation, random probing, and other curiositydriven approaches do, and ii) can be used for enhancing performance in multiple applications including distilling optimal planning to a policy net, collaboration, and competition. A video demo is available at https://www.dropbox.com/ s/8mz6rd3349tso67/Probing_Demo.mov?dl=0.
This paper presents a method for interactive agent modeling that involves learning to model a demonstrator agent not only through passively viewing the demonstrator agent, but also through interactions from a learner agent that learns to probe the environment of the demonstrator agent so as to maximally change the behavior of the demonstrator agent. The approximated demonstrator agent is trained through standard imitation learning techniques and the learning or probing agent is trained using reinforcement learning. The mind of the demonstrating agent is modeled as a latent space representation from a neural net. This latent space representation is used as the reinforcement learning signal for the learner (probing) agent similar to the curiosity driven techniques where larger changes in the representation of mind are sought out since they should lead to larger differences in demonstrator agent behavior. The authors test this in several gridworld environments as well as a sorting task and show that their method achieves superior performance and generalizes better to unseen states and task variations compared to several baseline methods. 
General comments, in no particular order:
1. The authors should provide more details on how the hand-crafted demonstrator agents were made. I assume something similar to an a* algorithm was probably used for the passing task, but what about the maze navigation task? 
2. The demonstrated tasks are (gridworld and algorithmic) which are very simple RL taks with low-dimensional (non-visual) state-spaces.  It's unclear how this would scale to more complex tasks with higher-dimensional state spaces such as Atari, Starcraft II or if this would work with tasks with continuous state and action spaces such as mujoco. 
3. The core premise behind training the learner agent with RL is using a curiosity driven approach to train a probing policy to incite new demonstrator behaviors by maximizing the differences between the latent vectors of the behavior trackers at different time steps. Because the latent vector is modeled as a non-linear function, distances between latent vector representations do not necessarily correspond to similar distances between behavior policies (for example, KL distances between two policy distributions). Since this is for ILCR, I think the authors should have taken a deeper dive into examining those latent representations and potentially visualizing those distances and how they correspond to different policy behaviors. 
4. The biggest flaw that I see in this method is the practicality of it's use. This method relies on the ability to obtain or gain access to a demonstration agent to learn from. In very simple tasks, such as the one presented here, the authors were able to hard-code their own demonstration agent. However, in harder tasks, this will not be feasible. If you are able to obtain or code your own agent, then you've already solved the task and there is no need to do any sort of imitation learning in the first place.  In reality, for sufficiently difficult tasks, a human would be the demonstration agent (as is done in most robotics tasks). In practice, imitation learning from a human works well since the learning can be done offline (i.e., post-hoc after a set of demonstrations are collected from the human). However, this task requires the learning to be interactive and thus the demonstrator needs to be present during the learning.  Interactively learning from a human becomes a problem if the learning takes tens of thousands of episodes of training since a human cannot reasonably be expected to be present for that amount of time. Thus, the question is 1) how well will this method work with a human acting as the demonstrator? and 2) how can this method work if you are not able to have access to a demonstrator long periods of time (or even at all)?
5. My previous comment relates mainly to the application of improved imitation learning. However, I do think this is still very useful in the context of multi-agent reinforcement learning for collaborative and competitive tasks (sections 4.6 and 4.7). I think this method demonstrates a method for improved collaborative and/or competitive performances given the fact that you already have a single agent with a learned policy. 
Overall, I think the paper presents a really nice idea of how to improve modeling of agents. essentially, a learner agent learns how to probe a demonstrator agent to provide more information about what's being demonstrated and prevent over-fitting to a set of fixed demonstrations.   This work sounds novel to me from a reinforcement learning perspective, however, I'm not well versed on theory of mind research.