Learning to Play With Intrinsically-Motivated, Self-Aware Agents
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to mathematically formalize these abilities using a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which an agent can move and interact with objects it sees, we propose a "world-model" network that learns to predict the dynamic consequences of the agent's actions. Simultaneously, we train a separate explicit "self-model" that allows the agent to track the error map of its worldmodel. It then uses the self-model to adversarially challenge the developing world-model. We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering. Moreover, the world-model that the agent learns supports improved performance on object dynamics prediction, detection, localization and recognition tasks. Taken together, our results are initial steps toward creating flexible autonomous agents that self-supervise in realistic physical environments.
The work presents a new approach on exploration yielding towards self-supervising agents. The proposed idea relies on learning two models, a world-model and a self-model. The former predicts the world dynamics while the latter predicts errors of the world-model and is used to select actions that challenge the current world-model. This enables the agent to develop a good world model.   The overall approach is interesting but as far as I understood, it delivers in the end another approach for exploration.   The applied metrics are not further specified than in the paragrapg starting in line 219, so the actual evaluation is unclear to me, e.g. I do not understand how exactly the appearence of interesting behaviors is quantified.  It is not clear to me how exactly the different phases are determined i.e. the onsets of 'Emergence of object attention', 'Object interaction learning' etc. It would also have been nice to see a video / simulation of the behavior over the whole period (or at least plots from important phases, similar to Figure 5).  Due to the two points mentioned above (and brevity in explanation), the experiments and the respective evaluations are a bit squishy for me and i can not grasp what is actually achieved. Especially in the part of task transfer (starting in line 286) it is not clear to me what exactly was done. For example, I do not know what exactly is supposed to be transferred and also the whole setup did not appear to me to actively solve tasks but to rather explore and learn models. Thus, I also do not understand what is presented in Table 1. The accuracy is computed on what?  In Figure 4, (c) and (d) have only 200 steps or is the addition (in thousands) missing? It is also not clear on how many trials every model was evaluated and thus, for (a) and (b) I'm not sure whether I am just looking at a smoothed curve plotted over the original values? For (c) and (d), it looks more definite like an error area, but it is not clear to me whether I see the variance, twice the standard deviation or whatever of how many evaluations? The same holds for Figure 6.  A problem of the presented action space representation may be that it depends on the number of objects available. This may become quite hard for messy and cluttered environments. Also, the highest amount of objects evaluated was N = 2, so it would be interesting to see how well the approach scales for fuller rooms and how the behavior develops.   Additionally, for me, having almost the same amount of supplementary as submitted pages is a bit too much and unfair towards other authors who are limiting themselves. Thus, an alternative option could be to publish the work as a journal article.