Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributions
We study the problem of online learning Markov Decision Processes (MDPs) when both the transition distributions and loss functions are chosen by an adversary. We present an algorithm that, under a mixing assumption, achieves O( T log |Π| + log |Π|) regret with respect to a comparison set of policies Π. The regret is independent of the size of the state and action spaces. When expectations over sample paths can be computed efficiently and the comparison set Π has polynomial size, this algorithm is efficient. We also consider the episodic adversarial online shortest path problem. Here, in each episode an adversary may choose a weighted directed acyclic graph with an identified start and finish node. The goal of the learning algorithm is to choose a path that minimizes the loss while traversing from the start to finish node. At the end of each episode the loss function (given by weights on the edges) is revealed to the learning algorithm. The goal is to minimize regret with respect to a fixed policy for selecting paths. This problem is a special case of the online MDP problem. It was shown that for randomly chosen graphs and adversarial losses, the problem can be efficiently solved. We show that it also can be efficiently solved for adversarial graphs and randomly chosen losses. When both graphs and losses are adversarially chosen, we show that designing efficient algorithms for the adversarial online shortest path problem (and hence for the adversarial MDP problem) is as hard as learning parity with noise, a notoriously difficult problem that has been used to design efficient cryptographic schemes. Finally, we present an efficient algorithm whose regret scales linearly with the number of distinct graphs.
Summary: On its first part the paper deals with 
      online learning of Markov Decision Processes. Given an MDP and a set of 
      policies (=distributions over actions) how should one Summary: The 
      first part of the paper addreses the problem of Online Learning in Markov 
      Decision Process. In this problem at each iteration: 1) adversary 
      chooses a loss function and transition kernel (probabilistic function that 
      chooses the next state based on the currenone and the selected action) 
      2) the leaner knows the current state x_t and chooses an action a_t. 
      3) the leaner sees the adversary choices and suffers the loss 4) 
      the next state is sampled. The goal of the learner is to minimize the 
      regret w.r.t. PI - a set of action selction policies The authors 
      propose an algorithm which is a variant of the Shrinking Dartboard 
      algorithm and show a bound of O((Tlog|PI|)^{1/2}). The second part 
      of the paper addreses the Adersarial Online Shortest Path Problem. Here at 
      each iteration the learner is presented with a graph with source, 
      destinations and all other nodes in layers structure. The Learner 
      chooses a path from source to dest. and then the weights of the edges is 
      revealed and the loss of the learner is defined as the sum of weights 
      along the chosen path. The goal of the learner is to minimize the 
      regret w.r.t. a set of path choosing policies. The main contribution 
      on this part is by showing that if both the graph and weights are chosen 
      by an adversary at each stage this problem is as hard as the online 
      agnostic parity problem (which is assumed to be a hard problem). 
      Quality & Clarity The results presented in the paper 
      seem new and interesting to me. However, 2 points bothered me during 
      the reading: 1) It would be helpful if the explanation of the 
      difference between the online MDP problem and standard online problems 
      (lines 200-204) will appear much earlier and would be a bit longer than 2 
      sentences. 2) The connection between the 2 parts of the paper is not 
      obvious to the learner. The authors state that Online Shortest Path is a 
      special case of MDP. I assume that this is correct but expect that such 
      claim will be accompanied by a short explanation (States are:..., Actions 
      are:... Loss function:...) (I haven't read the Supplementary material) 
      Originaluty & Significance I'm not familiar with previous 
      results on this problem. The hardness proof might be useful for other 
      works in the area Comments: 1) In Remark 3 the authors 
      claim that a bound for determintic policies won't apply a bound w.r.t. the 
      optimal policy. I'm not sure that I understand this claim: as far as I 
      know a regret of an algorithm is always measured w.r.t. a set of competing 
      policies. Hence, a bound on the regret of every deterministic policy will 
      imply an optimal regret w.r.t. any deterministic policy. I undrstand that 
      there might be some stochastic policy which is better but this policy is 
      not part of the competing group of policies. 2) In the hardness 
      results the authors mention several times "noisy parity" however, I didn't 
      find where the noise is mentioned. (Did you mean that the labeling y is 
      sometime flipped? Is it matters how is is flipped?) Typos: 
      Line 64 defines policy as mapping from actions intead of a mapping 
      from states (as in line 135) Figure1:OMDP algo: W_0 is undefined, 
      w_{pi_0},-1 is undefined. Line 178: This notion of regret w.r.t. 
      specific policy wasn't defined previously. (and appears a lot in the 
      paper) Line 367: A denotes an algorithm while earlier it was used for 
      the set of possible actions. Line 412: Did you define earlier N as the 
      set of all nodes? Line 412 at the end: shouldn't the l in Q_{pi, g l} 
      be changed to c?