EMI: Exploration with Mutual Information Maximizing State and Action Embeddings
Policy optimization struggles when the reward feedback signal is very sparse and essentially becomes a random search algorithm until the agent stumbles upon a rewarding or the goal state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or more ad-hoc measures of surprise. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show the state of the art performance on challenging locomotion task with continuous control and on image-based exploration tasks with discrete actions on Atari.
 This is a very interesting paper about a novel approach to exploration in agents with state and action representations, making heavy use of recent progress in the use of deep learning for estimating and maximizing mutual information, as well as introducing an approach to model the latent space dynamics with a linear models with sparse errors. A closely related work which is not mentioned is the work of Thomas et al 2017 arXiv:1708.01289 where they also maximize mutual information between distributed representations of actions (policies, actually) and of distributed representations of changes in the state (as the result of applying the policy). The phrase *functionally similar states* is used several times and would require a bit of explanation. I would also like to see more motivations for the two different reward functions r_e and r_d, and why one should be computed before the update while the other should be computed after. Regarding the experiments, and this is probably the weakest part of this paper, I would have expected to see comparisons against several of the numerous exploration methods which have been proposed in the past and are discussed in the paper (with many negative comments about their weakness, but no empirical support provided). Only one (EX2) was compared. The comparison with TRPO is without exploration (if I understand well, but should be stated clearly). It*s also not clear how these results compare to the best reported results on these games (whether or not exploration is used).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
EMI: Exploration with Mutual Information Maximizing State and Action Embeddings
Policy optimization struggles when the reward feedback signal is very sparse and essentially becomes a random search algorithm until the agent stumbles upon a rewarding or the goal state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or more ad-hoc measures of surprise. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show the state of the art performance on challenging locomotion task with continuous control and on image-based exploration tasks with discrete actions on Atari.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This is a very interesting paper about a novel approach to exploration in agents with state and action representations, making heavy use of recent progress in the use of deep learning for estimating and maximizing mutual information, as well as introducing an approach to model the latent space dynamics with a linear models with sparse errors. A closely related work which is not mentioned is the work of Thomas et al 2017 arXiv:1708.01289 where they also maximize mutual information between distributed representations of actions (policies, actually) and of distributed representations of changes in the state (as the result of applying the policy). The phrase *functionally similar states* is used several times and would require a bit of explanation. I would also like to see more motivations for the two different reward functions r_e and r_d, and why one should be computed before the update while the other should be computed after. Regarding the experiments, and this is probably the weakest part of this paper, I would have expected to see comparisons against several of the numerous exploration methods which have been proposed in the past and are discussed in the paper (with many negative comments about their weakness, but no empirical support provided). Only one (EX2) was compared. The comparison with TRPO is without exploration (if I understand well, but should be stated clearly). It*s also not clear how these results compare to the best reported results on these games (whether or not exploration is used).
assistant
 This is a very interesting paper about a novel approach to exploration in agents with state and action representations, making heavy use of recent progress in the use of deep learning for estimating and maximizing mutual information, as well as introducing an approach to model the latent space dynamics with a linear models with sparse errors. A closely related work which is not mentioned is the work of Thomas et al 2017 arXiv:1708.01289 where they also maximize mutual information between distributed representations of actions (policies, actually) and of distributed representations of changes in the state (as the result of applying the policy). The phrase *functionally similar states* is used several times and would require a bit of explanation. I would also like to see more motivations for the two different reward functions r_e and r_d, and why one should be computed before the update while the other should be computed after. Regarding the experiments, and this is probably the weakest part of this paper, I would have expected to see comparisons against several of the numerous exploration methods which have been proposed in the past and are discussed in the paper (with many negative comments about their weakness, but no empirical support provided). Only one (EX2) was compared. The comparison with TRPO is without exploration (if I understand well, but should be stated clearly). It*s also not clear how these results compare to the best reported results on these games (whether or not exploration is used).