Is the Bellman residual a bad proxy?
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximize the mean value, and derive a method that minimizes the residual T * v π − v π 1,ν over policies. A theoretical analysis shows how good this proxy is to policy optimization, and notably that it is better than its value-based counterpart. We also propose experiments on randomly generated generic Markov decision processes, specifically designed for studying the influence of the involved concentrability coefficient. They show that the Bellman residual is generally a bad proxy to policy optimization and that directly maximizing the mean value is much better, despite the current lack of deep theoretical analysis. This might seem obvious, as directly addressing the problem of interest is usually better, but given the prevalence of (projected) Bellman residual minimization in value-based reinforcement learning, we believe that this question is worth to be considered.
The paper investigates a fundamental question in RL, namely whether we should directly maximize the mean value, which is the primary objective, or if it is suitable to minimize the Bellman residual, as is done by several algorithms.  The paper provides new theoretical analysis, in the form of a bound on residual policy search.  This is complemented by a series of empirical results with random (Garnet-type) MDPs.   The paper has a simple take-home message, which is that minimizing the Bellman residual is more susceptible to mismatch between the sample distribution and the optimal distribution, compared to policy search methods that directly optimize the value.
The papes is well written; it outlines a specific question, and provides both theoretical and empirical support for answering this question.  The new theoretical analysis extends previous known results.  It is a modest extension, but potentially useful, and easy to grasp.  The empirical results are informative, and constructed to shed light on the core question.  I would be interested in seeing analogous results for standard RL benchmarks, to verify that the results carry over to "real" MDPs, not just random, but that is a minor point.
One weakness of the work might be its scope: it is focused on a simple setting of policy search, without tackling implication of the work for more complex settings more commonly used, such as TRPO, DDPG.