INFORMATION-DIRECTED EXPLORATION FOR DEEP REINFORCEMENT LEARNING
Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.
The authors propose a way of extending Information-Directed Sampling (IDS) to reinforcement learning. The proposed approach uses Bootstrapped DQN to estimate parametric uncertainty in Q-values, and distributional RL to estimate intrinsic uncertainty in the return. The two types of uncertainty are combined to obtain a simple exploration strategy based on IDS. The approach outperforms a number of strong baselines on a subset of 12 Atari 2600 games.
Clarity - I found the paper to be very well-written and easy to follow. Both the background material and the experimental setup were explained very clearly. The main ideas were also motivated quite well. It would have been nice to include a bit more discussion of why IDS is a good strategy, i.e. what are the theoretical guarantees in the bandit case? Section 3.2 could also provide a more intuitive argument.
Novelty - The paper essentially combines the IDS formulation of Kirschner & Krause, Bootstrapped DQN of Osband et al., and the C51 distributional RL method of Bellemare et al. Most of the novelty is in how to combine these ideas effectively in the deep RL setting, which I found sufficient.
Significance - Improving over existing exploration strategies for deep RL would be a significant achievement. While the results are impressive, I have a few concerns regarding some of the claims.
The subset of games used to evaluate the proposed approach seems to be biased towards games where there is either a dense reward or exploration is known to be easy. Almost every deep RL paper on exploration includes results for at least some of the hard exploration games (see “Unifying Count-Based Exploration and Intrinsic Motivation”). Why were these games excluded from the evaluation? The results would be much stronger if results on all 57 games were included.
The main difference between DQN-IDS and C51-IDS is that C51-IDS will tend to favor actions with lower return uncertainty. Doesn’t this mean that the improved performance of C51-IDS is due to an improved ability to exploit rather than explore? If this is indeed the case, then I would expect more evidence that this doesn't come at a cost of reduced performance on tasks where exploration is difficult.
Finally, the comparison between Bootstrapped DQN and DQN-IDS conflates the exploration strategies (IDS vs Thompson sampling) with the choice of optimizer (Adam vs RMSProp), so the claim that simply changing the exploration strategy to IDS leads to a major improvement is not valid. It would be interesting to see results for Bootstrapped DQN using the authors’ implementation and choice of optimizer to fully separate the effect of the exploration strategy.
Overall quality - This is an interesting paper with some promising results. I’m not convinced that the proposed method leads to better exploration, but I think it still makes a valuable contribution to the work on balancing exploration and exploitation in RL.
-------
The rebuttal and revisions addressed some of my concerns so I am increasing my score to 7