Published as a conference paper at ICLR 2018 DORA THE EXPLORER: DIRECTED OUTREACHING REINFORCEMENT ACTION-SELECTION
Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection. Exploration, however, can be more efficient if directed toward gaining new world knowledge. Visit-counters have been proven useful both in practice and in theory for directed exploration. However, a major limitation of counters is their locality. While there are a few model-based solutions to this shortcoming, a model-free approach is still missing. We propose E-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories. We compare our approach to commonly used RL techniques, and show that using E-values improves learning and performance over traditional counters. We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs. We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game.
The paper proposes an approach to exploration based on initializing a value function to 1 everywhere, then letting the value decay back toward zero as the state space is explored. I like the idea a lot. I don't really like the paper, though. I'd really like to see a strong theoretical and/or empirical justification for it, and both are lacking. On the theoretical side, can a bound be proven for this approach, even in the tabular case? On the empirical side, there are more (and more recent!) testbeds that have come to define the field---the mountain car problem is just not sufficient to convincingly argue that the method scales and generalizes. My intuition is that such an approach ought to be effective, but I really want to see additional evidence. Given the availability of so many RL testbeds, I worry that it had been tried but failed.
Detailed comments:
"Where γ is" -> ", <newline> where γ is".
"The other alternative" -> "The alternative"?
"without learning a model (Mongillo et al., 2014).": Seems like an odd choice for a citation for model-free RL. Perhaps select 
the paper that first used the term? Or an RL survey?
Right before Section 1.1, put a period after the Q-learning update equation.
"new states may" -> "new states, may".
"such approaches leads" -> "such approaches lead".
"they still fails" -> "they still fail".
"evaluated with respect only to its immediate outcome": Not so. Several of the cited papers use counters to determine which 
states are "known" and then solve an MDP to direct exploration past immediate outcomes.
" exploration bonus(Little & Sommer, 2014)" -> " exploration bonus (Little & Sommer, 2014)".
"n a model-free settings." -> "n model-free settings.".
" Therefore, a satisfying approach for propagating directed exploration in model-free reinforcement learning is still missing. ": I think you should cite http://research.cs.rutgers.edu/~nouri/papers/nips08mre.pdf , which also combines a kind of counter 
idea with function approximation to improve exploration.
"initializing E-values to 1": I like this idea. I wonder if one could prove bounds similar to the delayed Q-learning algorithm with 
this approach. It is reminiscent of https://arxiv.org/pdf/1205.2606.pdf , which also drives exploration by beginning with an 
overly optimistic estimate and letting the data (in a function approximation setting) decay the influence of this initialization.
"So after visited n times" -> "So after being visited n times".
"figure 1a" -> "Figure 1a". (And, in other places.)
"An important property of E-values is that it decreases over repetition" -> "An important property of E-values is that they decrease over repetition".
"t utilize counters, can" -> "t utilize counters can".
" hence we were interested a convergence measure": Multiple problems in this sentence, please fix.
Figure 2: How many states are in this environment? Some description is needed.
Figure 3: The labels in this figure (and all the figures) are absurdly small and, hence, unreadable.
"now turn to show that by using counters," -> "now turn to showing that, by using counters,".
Theorem 3.1: I'm not quite getting why we want to take a stochastic rule and make it deterministic. Note that standard PAC-MDP algorithms choose deterministically. It's not clear why we'd want to start from a stochastic rule.
" models(Bellemare" -> " models (Bellemare".
"Efficient memory-based learning for robot control": This reference is incomplete. (I'm skeptical that it represents the first use of this problem, but I can't check it.)
"Softmax exploration fail" -> "Softmax exploration fails".
"whom also analyzed" -> "who also analyzed".
"non-Markovity" -> "non-Markovianness"?