SOFT Q-LEARNING WITH MUTUAL-INFORMATION REGULARIZATION
We propose a reinforcement learning (RL) algorithm that uses mutual-information regularization to optimize a prior action distribution for better performance and exploration. Entropy-based regularization has previously been shown to improve both exploration and robustness in challenging sequential decision-making tasks. It does so by encouraging policies to put probability mass on all actions. However, entropy regularization might be undesirable when actions have significantly different importance. In this paper, we propose a theoretically motivated framework that dynamically weights the importance of actions by using the mutualinformation. In particular, we express the RL problem as an inference problem where the prior probability distribution over actions is subject to optimization. We show that the prior optimization introduces a mutual-information regularizer in the RL objective. This regularizer encourages the policy to be close to a nonuniform distribution that assigns higher probability mass to more important actions. We empirically demonstrate that our method significantly improves over entropy regularization methods and unregularized methods.
The authors take the control-as-inference viewpoint and learn a state-independent prior (which is typically held fixed). They claim that this leads to better exploration when actions have different importance. They relate this objective to a mutual information constrained RL objective in a limiting case. They then propose a practical algorithm, MIRL and compare their algorithm against DQN and Soft Q-learning (SQL) on 19 Atari games and demonstrate improvements over both.
Generally I found the idea interesting and at a high level the deficiency of entropy regularization makes sense. However, I had great trouble understanding the reasoning behind their method and did not find the connection to mutual information helpful. Furthermore, I had a number of questions about the experiments. If the authors can clarify their motivation and reasoning and strengthen the experiments, I'd be happy to raise my score.
In Sec 3.1, why is it sensible to optimize the prior? Can the authors give intuition for maximizing \log p(R = 1) wrt to the prior? This is critical for justifying their approach. Currently, the authors provide a connection to MI, but don't explain why this matters. Does it justify the method? What insight are we supposed to take away from that? 
The experiments could be strengthened by addressing the following:
* What was epsilon during training? Why was epsilon = 0.05 in evaluation? This is quite high compared to previous work, and it makes sense that this would degrade MIRLs performance less than DQN and SQL.
* What is the performance of SQL if we use \rho as the action selector in \epsilon-greedy. This would help understand if the performance gains are due to the impact on the policy or due to the changes in the behavior policy.
* Plotting beta over time
* Comparing the action distributions for SQL and MIRL to understand the impact of the penalty. In general, a deeper analysis of the impact on the policy is important. 
* Are their environments we would expect MIRL to outperform SQL based on your theoretical understanding? Does it?
* How many seeds were run per game?
* How and why were the 19 games selected from the full set?
Comments:
The abstract claims state-of-the-art performance, however, what is actually shown is that MIRL outperforms DQN and SQL.
With a fixed prior, the action prior can be absorbed into the reward (e.g., Levine 2018), so it is of no loss of generality to assume a uniform prior.
Could state that the stationary distribution is assumed to exist and be unique.
In Sec 3.1, why is the prior state independent?
In Sec 3.1, p(R = 1|\tau) is defined to be proportional to exp(\beta \sum_t r_t). Is this well-specified? How would we compute the normalizing constant since p(R = 0 | \tau) is not defined?
Throughout, I suggest that the authors not use the phrases "closed form" and "analytic" for expressions that are in terms of intractable quantities. 
It should be noted that Sec 3.2 Optimal policy for a fixed prior \rho follows from Levine 2018 and others by transforming the fixed prior into a reward bonus.
In Sec 3.2, the last statement does not appear to be necessary for the next subsection. Remove or clarify?
I believe that the connection to MI can be simplified. Plugging in the optimal \rho into Eq 3, we can see that Eq 3 simplifies to \max_\pi E_q[ \sum_t \gamma^t r_t] - (1 - gamma)/\beta MI_p(s, a) where p(s, a) = d^\pi(s) * \pi(a | s) and d^\pi is the discounted state visitation distribution. Thus Eq 3 can be thought of as a lower bound on the MI regularized objective.
In Sec 4, the authors state the main difference between their soft operator and the typical soft operator. What other differences are there? Is that the only one?
Sec 5 references the wrong Haarnoja reference in the first paragraph.
In Sec 5, alpha_beta = 3 * 10^5. Is that correct?
=====
11/26
At this time, the authors have not responded to the reviews. I have read the other reviews and comments, and I'm not inclined to change my score.
====
12/7
The authors have addressed most of my concerns, so I have raised my score. I'm still concerned that the exploration epsilon is quite different than existing work (e.g., https://github.com/google/dopamine/tree/master/baselines).