We study reinforcement learning algorithms with nonlinear function approximation in the online setting. By formulating both the problems of value function estimation and policy learning as bilevel optimization problems, we propose online Q-learning and actor-critic algorithms for these two problems respectively. Our algorithms are gradient-based methods and thus are computationally efficient. Moreover, by approximating the iterates using differential equations, we establish convergence guarantees for the proposed algorithms. Thorough numerical experiments are conducted to back up our theory.
The paper casts the problems of value learning and policy optimization, which can be problematic the non-linear setting, into the bilevel optimization framework. It proposes two novel algorithms with convergence guarantees. Although other works with similar guarantees exist, these algorithms are very appealing for their simplicity. A limited empirical evaluation is provided for the value-based method in Acrobot and Mountain Car and in the Atari games Pong and Breakout for the proposed bilevel Actor Critic.
There are a few missing references to similar, recent work, including Dai et al’s saddle-point algorithm (https://arxiv.org/pdf/1712.10285.pdf). Also, the claim that this is “the first attempt to study the convergence of online reinforcement learning algorithms with nonlinear function approximation” can’t be true (even replacing ‘attempt’ by ‘successfully’, there is e.g. Maei et al.’s nonlinear GTD paper, see below).
Although certainly interesting, the claims relating bilevel optimization and the target network are not completely right. E.g. Equation 3.6 as given is a hard constraint on omega. More explicitly, there are no guarantees that either network is the minimizer of the RHS quantity in 3.6.
The two-timescale algorithm is closer in spirit to the use of a target network, but in DQN and variants the target network is periodically reset, as opposed to what the presented theory would suggest. A different breed of “soft target” networks, which is more closely related to bilevel optimization has been used to stabilize training in DDPG (https://arxiv.org/abs/1509.02971).
There was some confusion for me on the first pass that you define two algorithms called ‘online Q-learning’ and ‘actor-critic’. Neither algorithm is actually that, and they should be renamed accordingly (perhaps ‘bilevel Q-Learning’ and ‘bilevel actor-critic’?). In particular, standard Q-Learning is online; and the actor-critic method does not minimize the Bellman residual (i.e. I believe the RHS of 3.8 is novel within policy-gradient methods).
Once we’re operating on a bounded space with continuous operators, Theorem 4.2 is not altogether surprising – a case of Brouwer’s fixed point theorem, short of the result that theta* = omega*, which is explained in the few lines below the theorem. While I do think Theorem 4.2 is important, it would be good to contrast it to existing results from the GTD family of approaches. Also, requiring that |Q_theta(s,a)| <= Qmax is a significant issue -- effectively this test fails for most commonly used value-based algorithms.
The empirical evaluation lacks any comparison to baselines and serves for little more than as a sanity check of the developed theory. This is probably the biggest weakness of the paper, and is unfortunate given the claim of relevance to e.g. deep RL.
Questions
Throughout, the assumption of the data being sampled on-policy is made without a clear argument as to why. Would the relaxation of this assumption affect the convergence results?
Can the authors provide an intuitive explanation if/why bilevel optimization is necessary?
Can you contrast your work with Maei et al., “Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation”?
Suggestions
The discussion surrounding the target network should be improved. In particular, claiming that the DQN target network can be viewed “as the parameter of the upper level optimization subproblem” is a stretch from what is actually shown.
The paper was sometimes hard to follow, in part because the claims are not crisply made. I strongly encourage the authors to more clearly relate their results to existing work, and ensure that the names they use match common usage.
I would have liked to know more about bilevel optimization, what it aims to solve, and the tools used to do it. Instead all I found was very standard two time-scale methods, which was a little disappointing – I don’t think these have been found to work particularly well in practice. This is particularly relevant in the context of e.g. the target network question.
A proper empirical comparison to existing algorithms would significantly improve the quality and relevancy of this work. There are tons of open-source baselines out there, in particular good state of the art implementations. Modifying a standard implementation to optimize its target network along the lines of bilevel optimization should be relatively easy.
Revision:
I thank the authors for their detailed feedback, but still think the work isn't quite ready for publication. After reading the other reviews, I will decrease my score from 6 to 5. Some sticking points/suggestions:
- Some of my concerns remain unanswered. E.g. the actor-critic method 3.8 is driven by the Bellman residual, which is not the same as e.g. the MSPBE used with linear function approximation. There is no harm in proposing variations on existing algorithms, and I'm not sure why the authors are reluctant to do. Also, Brouwer's fixed point theorem, unlike Banach's, does not require a contractive mapping.
- The paper over-claims in a number of places. I highly recommend that the authors make their results more concrete by demonstrating the implications of their method on e.g. linear function approximation. This will also help contrast with Dai et al., etc.