Near-optimal Reinforcement Learning in Factored MDPs
Any reinforcement learning algorithm that applies to all Markov decision processes (MDPs) will su er ( Ô SAT ) regret on some MDP, where T is the elapsed time and S and A are the cardinalities of the state and action spaces. This implies T = (SA) time to guarantee a near-optimal policy. In many settings of practical interest, due to the curse of dimensionality, S and A can be so enormous that this learning time is unacceptable. We establish that, if the system is known to be a factored MDP, it is possible to achieve regret that scales polynomially in the number of parameters encoding the factored MDP, which may be exponentially smaller than S or A. We provide two algorithms that satisfy near-optimal regret bounds in this context: posterior sampling reinforcement learning (PSRL) and an upper confidence bound algorithm (UCRL-Factored).
New regret bounds for the problem of online RL in factored MDPs have been presented. The bounds improve on the-state-of-the-art in generic  RL in terms of  dependency on the size of MDP assuming that there exists some known graphical structure for the problem.  For highly structured MDPs with local transitions this can lead to an exponential improvement in terms of dependency on the size of problem.Quality and clarityThe paper is clearly written. I also quickly checked the technical parts ( proof sketches) which seems  correct  to me. As for the clarity of results  I think the main trick to generalize the results of generic RL to the case of  factored MDP  have been clearly explained.   In regard to the quality of results,  I think  the idea of exploiting the graphical structure to improve the regret bounds of RL seems  a very promising idea to deal with large scale MDPs.   However some of the assumptions such as the one that requires the prior knowledge of  graphical  structure seems to be a bit restrictive, as it is recognized by the authors, and may limit the applicability of this result. The paper would benefit from a more in-depth discussion of this issue. In particular  a discussion on the possibility of combining the standard structure discovery algorithms with  UCRL or PSRL  would greatly add to the quality of the paper (also whether it is possible to prove similar regret bounds  in that case). Finally the paper claims that the result is "near-optimal" however no corresponding lower bound to support this claim is provided.Significance and noveltyThe problem of exploration-exploitation trade off in factored MDPs has been  extensively studied before in the RL literature . In that sense the idea of using the graphical structure to deal with large scale problems is not new. The main novelty  is that here  the bounds are expressed in terms of  accumulated regret, whereas the focus of those prior works are mostly on PAC exploration bounds. This seems a rather  incremental contribution since it is known that one can transform PAC complexity bounds to regret bounds and vice versa (See Jasch et al 10 and NIPS  11 workshop talk by Peter Auer). As for the novelty of the proof techniques,   the paper mainly relies on the  analysis of UCRL and PSRL and the main lemma to extend those results to the case of factored MDP is a standard algebraic trick.   Minor comments:line 140: The connection between X and S and A should be defined earlier ( before definition 3). line 298: where appropriate-> where it is appropriateline 308-309: M^* and M_k are equal in law : this is only true when we use the  true prior  (prior from which M^* is sampled) in Bayesian inference. If we use a wrong prior for M_k then they are not any more equal.