Non-delusional Q-learning and Value Iteration
We identify a fundamental source of error in Q-learning and other forms of dynamic programming with function approximation. Delusional bias arises when the approximation architecture limits the class of expressible greedy policies. Since standard Q-updates make globally uncoordinated action choices with respect to the expressible policy class, inconsistent or even conflicting Q-value estimates can result, leading to pathological behaviour such as over/under-estimation, instability and even divergence. To solve this problem, we introduce a new notion of policy consistency and define a local backup process that ensures global consistency through the use of information sets-sets that record constraints on policies consistent with backed-up Q-values. We prove that both the model-based and model-free algorithms using this backup remove delusional bias, yielding the first known algorithms that guarantee optimal results under general conditions. These algorithms furthermore only require polynomially many information sets (from a potentially exponential support). Finally, we suggest other practical heuristics for value-iteration and Q-learning that attempt to reduce delusional bias.
UPDATE: Thank you to the authors for a well thought out response.  I definitely encourage you to compile the list of counterexamples that are or are not the result of delusion.  I think it will be a very important list for the theory community.  Also, thanks for the discounting example, that helped.  Good point on the Boyan example and ADP divergence and the expanded discussion.  Summary:   The paper defines the concept of ``delusional bias’’, the fact that the Bellman operator (or the sample-based version in Q-learning) combined with a function approximator, can backup values from a policy that exist in the realizable space of the approximator, essentially assuming that two actions that it cannot simultaneously take can be chosen at the same time. The authors propose a new operator that uses a partitioning of the parameter space into information sets.  The operator essentially maintains a large set of potential combinations of the partitions and q values for each of the sets, eventually choosing the maximum after convergence.  This requires significant (but sometimes polynomially bounded) computation, although in many cases an intractable witness procedure is needed for complex function approximators.  Review:  This is a potentially ground-breaking paper.  I have comments below on ways to improve the presentation of the paper and some minor mathematical notes but overall, I think the authors did an excellent job on a long standing foundational problem in RL.  The paper cites some important foundational work in the introduction but does not explicitly state which existing negative function approximation results are explained by delusional bias.  Is Baird’s counterexample a result of delusional bias and therefore solved by the new algorithm?  It would be better to explicitly mention which of the classical results can (and cannot) be explained by this phenomenon.  I do not see the direct link between delusional bias and the “discounting paradox” described in the paper.  The example given mixes a discounted MDP with an undiscounted one, and really those two problems are fundamentally different.  Is there an example where both discounts are < 1?  If so, is the conjecture of the paper that small discount factors essentially truncate the backups and therefore somehow enforce a partitioning?  I don’t understand why that would be the case.  Clearly changing the discount factor changes the solution, but it is not clear why the paradox is really related to delusional bias.    I thought perhaps the most important line in the paper was on line 235 where the authors note that their result is actually consistent with Petrik’s finding on the NP-Hardness of finding the best function approximator.  The authors essentially point out that Petrik was considering all possible action choices, and therefore maybe asking the wrong question.  Perhaps minimizing Bellman error isn’t all that important if one is considering actions that can’t actually be taken.  Perhaps that can be spelled out better here.  To make room for some of the explanations above, I suggest cutting down much of the speculation in the section 5.  These are interesting conjectures, but do not have any real empirical evidence behind them. If a particular example of one of these three approaches could be demonstrated here it would certainly improve the paper.  In section 3.2, it would be good to point out explicitly that there are different consequences of delusion in q-learning versus ADP.  Specifically, q-learning can actually diverge while I think the claim here is that ADP converges, but to something non-optimal given the approximator constraints, right?  That point should be made explicit.  Line 182 – the parentheses around the full sentence can be dropped.  It’s a full idea / sentence.