Exploration in Structured Reinforcement Learning
We address reinforcement learning problems with finite state and action spaces where the underlying MDP has some known structure that could be potentially exploited to minimize the exploration rates of suboptimal (state, action) pairs. For any arbitrary structure, we derive problem-specific regret lower bounds satisfied by any learning algorithm. These lower bounds are made explicit for unstructured MDPs and for those whose transition probabilities and average reward functions are Lipschitz continuous w.r.t. the state and action. For Lipschitz MDPs, the bounds are shown not to scale with the sizes S and A of the state and action spaces, i.e., they are smaller than c log T where T is the time horizon and the constant c only depends on the Lipschitz structure, the span of the bias function, and the minimal action sub-optimality gap. This contrasts with unstructured MDPs where the regret lower bound typically scales as SA log T . We devise DEL (Directed Exploration Learning), an algorithm that matches our regret lower bounds. We further simplify the algorithm for Lipschitz MDPs, and show that the simplified version is still able to efficiently exploit the structure.
*Summary* This paper deals with undiscounted reinforcement learning. It provides problem-related (asymptotic) lower and upper bounds on the regret, the latter for an algorithm presented in the paper that builds on Burnetas and Katehakis (1997) and a recent bandit paper by Combes et al (NIPS 2017). The setting assumes that an "MDP structure" \Phi (i.e. a set of possible MDP models) is given. The regret bounds (after T steps) are shown to be of the form K_Phi*log T, where the parameter K_\Phi is the solution to a particular optimization problem. It is shown that if \Phi is the set of all MDPs ("the unstructured case") then K_\Phi is bounded by HSA/\delta, where H is the bias span and \delta the minimal action sub-optimality gap. The second particular class that is considered is the Lipschitz structure that considers embeddings of finite MDPs in Euclidian space such that transition probabilities and rewards are Lipschitz. In this case, the regret bounds are shown to not to depend on the size of state and action space anymore.   *Evaluation* In my opinion this is an interesting theoretical contribution to RL. Although the analysis is somewhat restricted (ergodic MDPs, asymptotic analysis) and the paper considers only Lipschitz structures, I think the presented results are an important first step in a promising research direction. I did not have time to check the appendix, and the main part does not present any proof sketches either, so I cannot judge correctness. The paper is well-written, if a bit technical. A bit more intuition on how the algorithm works would have been welcome. Overall, in my opinion this is a clear accept.      *Comments* - In footnote 4, I think the uniqueness is about the optimal policy, not the bias span. - Algorithm: The role of the quantities s_t(x) and the parameters \epsilon, \gamma should be explained. Also, I wondered whether the minimization in the Exploit step is necessary, or whether an arbitrary element of \mathcal{O} could be chosen as well. An explanation of the difference of the Estimate and Explore steps would be helpful as well.   - In l.272 it is claimed that any algorithm for undiscounted RL needs to recompute a new policy at each step. This is not true. Actually, e.g. UCRL and similar algorithms wouldn't work if policies were recomputed at each step. - In the last line "unknown probabilistic rewards" are mentioned as future work, which I did not understand as the current paper (as far as I understood) already deals with such rewards.  *Typos and Minor Stuff* - l.112: Footnote 2 is placed unfortunate.  - l.142: "state"->"states" - l.154: "algorithms"->"algorithm" - eq.(2b): Delete dot in formula, replace comma by dot. - l.186: "are"->"is" - l.196: "Ee"->"We" - l.203: "makes"->"make" - l.205: Delete "would". - l.225: D might be confused with the diameter. - l.239: "model"->"models" - l.250: "a MDP"->"an MDP" - l.253: "on optimal"->"of the optimal" - l.256: "apparent best"->"apparently best" - l.260: "phase"->"phase)" - l.265: "model"->"model structure" - l.278: "an linear programming"->"a linear programming problem" - l.280: Delete "a". - l.282: "algorithm"->"algorithms" - l.284: "Works"->"Work" - l.300: "An other"->"Another" - l.319: Capitalize "kullback-leibler".  PS: Thanks for the detailed comments in the feedback.