Online Learning in Episodic Markovian Decision Processes by Relative Entropy Policy Search
We study the problem of online learning in finite episodic Markov decision processes (MDPs) where the loss function is allowed to change between episodes. The natural performance measure in this learning problem is the regret defined as the difference between the total loss of the best stationary policy and the total loss suffered by the learner. We assume that the learner is given access to a finite action space A and the state space X has a layered structure with L layers, so that state transitions are only possible between consecutive layers. We describe a variant of the recently proposed Relative Entropy Policy Search algorithm and show that its regret after T episodes is 2 L|X ||A|T log(|X ||A|/L) in the bandit setting and 2L T log(|X ||A|/L) in the full information setting, given that the learner has perfect knowledge of the transition probabilities of the underlying MDP. These guarantees largely improve previously known results under much milder assumptions and cannot be significantly improved under general assumptions.
Summary The paper is about online learning in 
      non-stationary episodic loop-free Markov Decision Processes, where the 
      loss function is allowed to change between episodes. The proposed 
      algorithm O-REPS is an online variant of the Relative Entropy Policy 
      Search algorithm, that searches for better policies by bounding the 
      related information loss, so that small policies updates are performed. 
      The authors describe the proposed algorithm and produce upper bounds 
      to the total expected regret both for the bandit and for the full feedback 
      cases, concluding that, in many common settings, O-REPS has better 
      guarantees than previous results from literature and requires less 
      assumptions on the MDP and puts no restriction to the class of policies 
      that must be considered. Quality The paper is technically 
      sound. The problem is well introduced and the main results are presented 
      with proofs (I have checked them and they seem correct). In the last 
      section, the authors compare the properties of the proposed approach 
      w.r.t. the ones of state-of-the-art algorithms, highlighting the 
      improvements introduced. Although no limitation of the proposed 
      approach is discussed, some ideas for future research directions are 
      suggested. Clarity The paper is well written and clearly 
      organized. The proofs are quite detailed, so that they can be easily 
      followed in most cases. I suggest only to add some justification to 
      inequalities in line 353. The only part that I found not very clear is 
      Section 5.3 where the authors tries to compare the proposed approach with 
      the original REPS algorithm. Overall, although the paper is quite 
      technical, I think that it can be followed and mostly understood also by a 
      non-expert audience. Originality The paper builds upon on a 
      state-of-the-art algorithm, but it considers a different setting and 
      produce a theoretical analysis of the algorithm properties that, as far as 
      I know, constitutes an original contribution. The authors compare the 
      obtained results with the ones obtained by other approaches, giving a 
      quite comprehensive overview of the related literature. The paper is 
      not groundbreaking, but represents a solid contribution that advances 
      knowledge in a relevant area of machine learning. Significance 
      The theoretical results reported in this paper represent a clear 
      improvement over the state of the art. The impact of such theoretical 
      results on future research can be hardly assessed due to the specific 
      class of MDPs addressed (episodic and loop-free). On the other hand, it 
      provides interesting results that may be useful to produce new studies in 
      more general settings. *** Comments after the rebuttal *** 
      Clarifications provided by author rebuttals will further improve 
      paper quality.