(More) Efficient Reinforcement Learning via Posterior Sampling
Most provably-efficient reinforcement learning algorithms introduce optimism about poorly-understood states and actions to encourage exploration. We study an alternative approach for efficient exploration: posterior sampling for reinforcement learning (PSRL). This algorithm proceeds in repeated episodes of known duration. At the start of each episode, PSRL updates a prior distribution over Markov decision processes and takes one sample from this posterior. PSRL then follows the policy that is optimal for this sample during the episode. The algorithm is conceptually simple, computationally efficient and allows an agent to encode prior knowledge in a natural way. We establish anÕ(τ S √ AT ) bound on expected regret, where T is time, τ is the episode length and S and A are the cardinalities of the state and action spaces. This bound is one of the first for an algorithm not based on optimism, and close to the state of the art for any reinforcement learning algorithm. We show through simulation that PSRL significantly outperforms existing algorithms with similar regret bounds.
Summary: this paper gives a regret analysis for an 
      episodic reinforcement learning motivated by posterior sampling (as known 
      as Thompson sampling). The idea is to sample, from the posterior, an MDP 
      model at the beginning of each episode, follow this MDP's optimal policy 
      in that episode, update the model posterior, and then repeat. In the past, 
      despite good empirical performance, this algorithm (called PSRL) remains a 
      heuristic approach without strong theoretical guarantees. This paper 
      provides the first regret analysis, with a bound of O(\tau * S * \sqrt{A * 
      T}), where H is the length of the episode and T is the total number of 
      stpes. Numerical experiments on very simple MDPs show PSRL significantly 
      outperform the other algorithm UCRL2 that enjoys a similar regret bound. 
      * Quality: this paper makes an interesting theoretical 
      contribution to RL in finite-state, finite-action, episodic MDPs. Building 
      on recent advances in the multi-armed bandit literature, it is a 
      successful step towards understanding finite-sample performance of 
      Bayesian-style RL algorithms. * Clarity: the paper is written very 
      clearly. Despite the heavy technical details, the basic ideas are 
      explained well and intuitively. * Originality: The algorithm PSRL 
      is not new; instead, the novelty in the paper is in its (first) regret 
      analysis. The key in the analysis is Lemma 2, which equates the *expected* 
      regret with the regret in the sampled MDP (assuming the MDP is sampled 
      from the posterior). A similar observation has been used to proved 
      expected regret bounds of posterior sampling in multi-armed bandits [15]. 
      While the rest of the analysis seems pretty standard in reinforcement 
      learning, the application of Lemma 2 in this kind of analysis, and 
      especially for expected regret, appears sufficiently novel. * 
      Significance: Posterior sampling has been very successful in multi-armed 
      bandits that raised quite a lot of interests in its theoretical study 
      recently. This paper is the first that extends analytic ideas to MDPs and 
      shows strong regret bounds of posterior sampling. Not only is the result 
      the first of its kind, it may generate further interests in the RL 
      community to investigate a new class of algorithms based on posterior 
      sampling. Detailed comments: * While the analysis and 
      bounds are very interesting, the paper should make it clear that the 
      nature of the bounds here is fundamentally different from that in the 
      literature. In particular, existing bounds are worst-case bounds, while 
      the bounds here are average-case (averaged by the prior/posterior). It is 
      the averaging that makes the key Lemma 2 possible. For the same reason, it 
      seems the paper's *expected* bounds may be weaker than worst-case bounds. 
      I hope the authors can comment on these points in the paper. * In 
      the paper, UCRL2 is the only algorithm to compare against PSRL. 
      Justifications are needed for this choice. PSRL, in the current 
      description in the paper, applies to episodic tasks and aims to optimize 
      undiscounted, finite-horizon total reward. In contrast, UCRL2 applies to 
      continuing tasks and aims to optimize average reward. As a consequence, 
      their regret bounds are not directly comparable: UCRL2 depends on a 
      diameter parameter, while PSRL has the horizon length in the bound. There 
      may be a way to relate them, but it is not obvious how. * Instead 
      of UCRL2, another piece of work may be more relevant: Claude-Nicolas 
      Fiechter: Efficient Reinforcement Learning. COLT 1994: 88-97 Although 
      the Fiechter paper considers the PAC framework, it considers 
      finite-horizon problems, so seems relevant to the present work. * 
      The simple experiments show PSRL is empirically much better than UCRL2. It 
      does not strike me as a surprise, given the very conservative nature of 
      UCRL2. And, because of the gap the kinds of problems the two algorithms 
      are designed for (see a related comment above), the empirical comparison 
      carries even less information. * Line 255, 'depends only the' 
      --> 'depends only on the' * Line 278, please be specific what 
      the 'high probability' is. * Line 411, is it too restrictive to 
      assume that the optimal average reward is uncorrelated with episodic 
      length?