Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes
While designing the state space of an MDP, it is common to include states that are transient or not reachable by any policy (e.g., in mountain car, the product space of speed and position contains configurations that are not physically reachable). This results in weakly-communicating or multi-chain MDPs. In this paper, we introduce TUCRL, the first algorithm able to perform efficient exploration-exploitation in any finite Markov Decision Process (MDP) without requiring any form of prior knowledge. In particular, for any MDP with S C communicating states, A actions and
C is the diameter (i.e., the length of the longest shortest path between any two states) of the communicating part of the MDP. This is in contrast with existing optimistic algorithms (e.g., UCRL, Optimistic PSRL) that suffer linear regret in weakly-communicating MDPs, as well as posterior sampling or regularised algorithms (e.g., REGAL), which require prior knowledge on the bias span of the optimal policy to achieve sub-linear regret. We also prove that in weaklycommunicating MDPs, no algorithm can ever achieve a logarithmic growth of the regret without first suffering a linear regret for a number of steps that is exponential in the parameters of the MDP. Finally, we report numerical simulations supporting our theoretical findings and showing how TUCRL overcomes the limitations of the state-of-the-art.
UPDATE: I have read the authors' response and it was very helpful and addressed my concerns well and I have adjusted my score.  I now see the flaw in the regal reasoning and encourage the authors to add the explicit callout of the backwards inequality in the supplementary material (unless I missed it).  I also agree with the reasons for not comparing to regal empirically.  The point about algorithm mechanics is well thought out, though I still see some benefit to calling it out.  The wording correction you mentioned for line 159 makes sense as well.   Summary:  The paper considers the problem of RL exploration/exploitation in weakly communicating MDPs where many states (S_T) may not actually be reachable.  Such states cause an overestimate of values in algorithms like UCRL, which consider such states in their contribution to the exploration bonus.  The authors propose a new algorithm, TUCRL, that performs an online estimate of the reachability of states, and discards states that are below a threshold based on visits and the number of steps so far.  The algorithm is shown to have improved regret in such MDPs.  Furthermore, a new lower bound is proven showing that no algorithm  in this setting can achieve logarithmic regret without first incurring  exponential regret.  Review:  Overall, the paper makes a strong case for TUCRL.  The proofs are thorough and convincing.  I appreciate having the sketches actually in the paper in addition to the complete results in the supplementary material.  The idea appears to be sound, since UCRL was not making use of the tighter bounds that can be derived on state reachability, and the empirical results verify that the real-world advantages of the improved bound are significant.  The lower bound is also an important result outside of the TUCRL analysis.  My only concerns are in the comparison to Regal, some of the terminology in Section 2, and the description of the algorithm mechanics but I think these can be easily addressed.  The paper’s claim of a flaw in the proof of Regal’s regret was surprising and the logic laid out in the supplementary material appears correct, but the explanation is worded in a confusing way that makes it unclear what exactly the flaw is.  On line 417 of the supplemental material, we see the flawed inequality -C \Sum{v} <= -C 2^j.  The authors then state: “[that inequality] is not true since -C := -c / sqrt(2^j).  However, that definition of C does not actually invalidate the inequality since C appears on both sides.  The authors later point out that what is needed is a lower bound on v, but they are unable to find one so they declare the proof incorrect.  All the evidence here is pointing to there being a flaw and the authors being right, but we’re missing an actual example where the inequality fails to hold.  The absence of a proof does not mean the statement is necessarily wrong (even though I believe it is).  Can the authors give us an actual MDP or even just plausible numerical instantiations of this inequality that make it untrue?  If so, that would seem to be a much cleaner refute of Regal.  In addition to the unclear terminology above, I was surprised that Regal did not appear in the experiments since it seems to be built for these kinds of environments, even if the theory is flawed.  Would an empirical result actually show TUCRL outperforming Regal?  If all we have is questions about its proof then an empirical result seems needed.  Most of the paper’s terminology is rigorously defined, but I found the definition of S^C and S^T in the beginning of Section 2 surprisingly informal.  The initial definitions make it unclear what cases are allowed under these conditions – are they hard partitions where only one is visited – can there be many “island” MDPs that don’t communicate but an episode can start in any one of them?  In that case, what counts as S^C?  Many of these questions are cleaned up laterin the text, particularly in the multi-chain MDP subsection later in section 2, but they should really be addressed with a formal and rigorous definition of S^C and S^T right at the start.  One of the most interesting parts of the proof was the bounding of the number of times a poorly visited (s,a) is visited.  This is reminiscent of escape probabilities in the PAC literature but in TUCRL, we see a UCB/UCRL like behavior where “jumps” in the exploration occur periodically (though with longer and longer periods).  This is also borne out in the empirical results.  I would have liked to have seen this intuition presented when the algorithm was introduced, maybe as a set of “key properties” of the algorithm so that users will better understand the behavioral characteristics of the algorithm.  Right now those details are scattered in various sections.  I realize space is tight, but you could add a few lines to the supplemental material covering this.  The main result is a good mathematical derivation but the significance of the empirical results put the theory in a better light.  The lower bound is also a strong contribution and an important component for a theory paper.   Minor notes:   Line 159: poorly visited so far are enabled -> This is unclear.  What does enabled mean?  Shouldn’t poorly visited states be discarded?