Improving PAC Exploration Using the Median of Means
We present the first application of the median of means in a PAC exploration algorithm for MDPs. Using the median of means allows us to significantly reduce the dependence of our bounds on the range of values that the value function can take, while introducing a dependence on the (potentially much smaller) variance of the Bellman operator. Additionally, our algorithm is the first algorithm with PAC bounds that can be applied to MDPs with unbounded rewards.
The paper proposes a PAC algorithm for reinforcement learning in MDPs with potentially finite state and action spaces. The key difference to existing work is that this method uses median of means instead of the usual empirical mean to estimate the expected value of the next state in the Bellman operator. This allows to provide PAC bounds (in the form of TCE bounds) that only depend log-linearly on the range of the value function while introducing dependency on the variance of the values of successor states.
The authors' feedback clarified misunderstandings I had and addressed my concerns about usefulness of the presented results. With the remaining issues with clarity being removed, this paper can be a really nice contribution for the NIPS community.    Original Review: ---------------- The idea of using the medians of means estimator to obtain stronger PAC guarantees for reinforcement learning algorithms seems very appealing to me and the presented results appear plausible. Yet, I see two main issues with this paper: First, there is a lack of clarity and insufficient explanations and second, the usefulness of the presented  PAC guarantee is not entirely clear to me.  Usefulness: * The authors state in line 40 on page 2 that their algorithm does not depend on the number of successor states which can even be unbounded. However, the sample complexity bound depends linearly on the number of states. For unbounded number of successor states and therefore unbounded number of states, the sample complexity bound becomes meaningless. * Definition 2.1 states that Q_max is an upper bound on the value of any state (for any policy I assume) but no assumptions have been made about a lower bound. I therefore wonder whether adding a constant to every reward (which should not affect the difficulty of the RL problem) affects the bound in Theorem 1 or not. Is it assumed that rewards / all values are nonnegative? * Definition 5.1 requires sigma^2 to be an upper bound on the variance of the value of the next state for any value-function and policy. This seems rather demanding and raises the question when sigma^2 is actually  significantly smaller than (Q_max/2)^2 especially discount factors close to 1. For gamma close to 1, it is sufficient that there is a (s,a) pair and a subset S' of S so that p(s' in S' | s, a) \approx 1/2 for sigma to be close to (Q_max/2)^2 (by choosing \tilde Q and pi appropriately). This does seem to be the case in most relevant MDPs.  Clarity and presentation: * The paper claims the algorithm is PAC-optimal (w.g. line 200). However, a direct comparison to a lower-bound is not provided which makes it not clear to me why this algorithm has indeed optimal (= matching a lower bound up to constants and log-terms) sample-complexity. * Knowing a nontrivial bound on sigma is unrealistic, especially ahead of executing the algorithm. However, the current bound assumes parameters k, k_m to be chosen depending on sigma. The bound in Theorem 5.2 is therefore rather artificial. It is stated in line 199 that the bound can also be formulated for the more relevant case of choosing k, k_m fixed but not actually provided in the paper or appendix.  * The quantities k_m, k and epsilon_s are only defined implicitly (e.g. k_m appears on both sides of the equality sign. It is therefore hard to identify how changes in problem characteristics affect the provided sample complexity bound. These dependencies should be disentangled to make the dependency of the bound on the important quantities: 1/(1-gamma), |S|, |A|, Q_max and sigma clear. * The provided proof sketch discusses the general structure which is very similar to the analysis of most other Optimism-in-the-face-of-uncertainty methods. However, it provides no insight where the median of means actually changes the proof and allows (potentially) tighter bounds. While Section 3 is very helpful for understanding why median of means can be beneficial, more details on how this is leveraged in PAC RL would be very much appreciated.  Minor things: * Should the value of of n in line 110.5 not depend on 2/delta instead of 1/delta as each of the two failure cases has to be bounded by 1- delta/2? * Algorithm 1: \epsilon_a is also an input / parameter of the algorithm * Line 20 of Algorithm 1 redefines i in the sum, which is already defined in line 16