"How hard is my MDP?" The distribution-norm to the rescue
In Reinforcement Learning (RL), state-of-the-art algorithms require a large number of samples per state-action pair to estimate the transition kernel p. In many problems, a good approximation of p is not needed. For instance, if from one state-action pair (s, a), one can only transit to states with the same value, learning p(·|s, a) accurately is irrelevant (only its support matters). This paper aims at capturing such behavior by defining a novel hardness measure for Markov Decision Processes (MDPs) based on what we call the distribution-norm. The distributionnorm w.r.t. a measure ν is defined on zero ν-mean functions f by the standard variation of f with respect to ν. We first provide a concentration inequality for the dual of the distribution-norm. This allows us to replace the problem-free, loose || · || 1 concentration inequalities used in most previous analysis of RL algorithms, with a tighter problem-dependent hardness measure. We then show that several common RL benchmarks have low hardness when measured using the new norm. The distribution-norm captures finer properties than the number of states or the diameter and can be used to assess the difficulty of MDPs.
Note to authors: As I stated in my original review, though I think it couldbe improved upon (and what paper couldn't?), I believe this paper should be accepted. My apologies for not making that more clear. I have edited my review below in an attempt to clarify some of my comments that were perhaps not stated carefully enough originally. ---Summary of Paper---The paper presents a novel measure of "hardness" for MDPs (based on anovel concentration inequality) which essentially measures thestochasticity of the *value* of the next state. Implications forperformance bounds for RL algorithms are sketched out. Empirically thenew measure is demonstrated to be tighter than previously knownindicators of MDP hardness.---Quality---The paper appears to be technically sound. The main results aretheoretical in nature and are well-supported. Due to space constraints, Section 3.1 comes across as rushed andincomplete. I understand the authors' excitement about the potentially broad implications of their result, but unfortunately there is simplynot enough room to fully roll those implications out. As a result,the paper currently presents some key implications (the newsimulation lemma, the modified UCRL bound), but leaves a lot offollow through as exercise for the reader (see lines 288-292, 311-314, 315-317, 338-341). I certainly couldn't expect the authorsto fill in all those details in the available space, and I agree that someof what they hint at is straightforward given sufficiently intimatefamiliarity with the relevant literature. Nevertheless, in my opinion,the presentation of this section leaves the reader feeling unresolved,and, depending on how much their intuition overlaps with the authors,perhaps unsure of whether the various paths the authors point to willreally be as smooth as they suggest. I wonder if the paper might havebeen better served by thoroughly and concretely rolling out oneimpactful consequence with hints in other directions, rather than thesample platter of incomplete offshoots that is presented here.The empirical results are helpful and evocative, and do support thecase that the new measure is tighter than other known measures. On theother hand, they stop short of demonstrating a more concreteconnection between the bounds given and learningperformance. A more thorough evaluation might have measured if the distinctions made by this quantity (some examples have notably higher C than others) are reflected in practice, or included some examples thought to be more difficult to show that the measure responds accordingly. Also a variation on UCRL is presented with an improved regret bound due tothe result in this paper, but whether this also results in betterperformance in practice is not investigated. The authors rightly pointout in their response that they make no claim to improve practicalperformance, and I suppose therefore they are not beholden to demonstratepractical impact. Nevertheless, this is the question we ultimately careabout, is it not? So it seems like it would have been worthwhile toinvestigate/report, regardless of the outcome.---Clarity---I found the paper to be by and large clear, readable, andwell-written. I appreciated the authors' efforts to continuallyconnect technical statements/results with high level ideas and goals.Continuing on my comments above, I will say that I feel the clarity ofSection 3 is harmed by the sheer number of points to be made. Spaceconstraints being what they are, this section moves at a rapid clipand, as mentioned, leaves a lot as "exercise for the reader." I fear thatreaders who are not as intimately familiar with the surrounding theoreticalcontext as the authors may not so readily leap to the conclusions thatthe authors intend to communicate.A couple of minor suggestions:- On lines 19-21 the abstract contains a very nice intuitive example  of when the new bound can be much tighter. That example does appear  in the main text, but not until page 5, and even then it is  expressed in more technical terms. I'd recommend adding this  intuition to the introduction as well. I think it is useful for the  reader to have it in their head as they work through the results,  and the abstract should not be relied upon for the flow of the  paper.- I'd humbly recommend that the authors give their measure a  name. Just writing this review it became clear how cumbersome it is  to discuss a nameless quantity. It may also have a positive impact  on readers' retention/understanding to have something more compact  and more evocative way to refer to this thing than C^\pi_M.---Originality---As far as I know the main results are novel and I agree with theauthors that their bound likely represents a new level of granularityin formally characterizing the "hardness" of learning MDPs. The workfeels sufficiently contextualized in the relevant literature.---Significance---I think there is clear theoretical/scientific significance to theresult. The authors are correct to point out the often wide gapbetween theoretical guarantees and practical experience. If we areever to gain a formal understanding of the phenomena we actuallyobserve (and if theoretical guarantees are ever going to be meaningfulto practitioners) results like this one, that examine problems at afar finer granularity, will be required. It's not clear that thisresult closes this gap, but I think it is compellingly demonstratedthat it takes a step in the right direction.At the moment the practical significance of the result is unclear. Thepaper does show that modifications to algorithms may be suggested bythe result, but it was not investigated whether the modification hadan impact in practice. I do agree with the authors that there is very real *potential* forthis result to have wide-reaching consequences both for the analysisof existing algorithms and for the development of new ones. Though insome ways I feel the paper could benefit from being retooled, in the end, I think that the potential for significant follow-up results overrides my concerns about presentation.