Deep reinforcement learning algorithms that estimate state and state-action value functions have been shown to be effective in a variety of challenging domains, including learning control strategies from raw image pixels. However, algorithms that estimate state and state-action value functions typically assume a fully observed state and must compensate for partial or non-Markovian observations by using finite-length frame-history observations or recurrent networks. In this work, we propose a new deep reinforcement learning algorithm based on counterfactual regret minimization that iteratively updates an approximation to a cumulative clipped advantage function and is robust to partially observed state. We demonstrate that on several partially observed reinforcement learning tasks, this new class of algorithms can substantially outperform strong baseline methods: on Pong with single-frame observations, and on the challenging Doom (ViZDoom) and Minecraft (Malmö) first-person navigation benchmarks.
This paper presents Advantage-based Regret Minimization, somewhat similar to advantage actor-critic with REINFORCE.
The main focus of the paper seems to be the motivation/justification of this algorithm with connection to the regret minimization literature (and without Markov assumptions).
The claim that ARM is more robust to partially observable domains is supported by experiments where it outperforms DQN.
There are several things to like about this paper:
- The authors do a good job of reviewing/referencing several papers in the field of "regret minimization" that would probably be of interest to the ICLR community + provide non-obvious connections / summaries of these perspectives.
- The issue of partial observability is good to bring up, rather than simply relying on the MDP framework that is often taken as a given in "deep reinforcement learning".
- The experimental results show that ARM outperforms DQN on a suite of deep RL tasks.
However, there are also some negatives:
- Reviewing so much of the CFR-literature in a short paper means that it ends up feeling a little rushed and confused.
- The ultimate algorithm *seems* like it is really quite similar to other policy gradient methods such as A3C, TRPO etc. At a high enough level, these algorithms can be written the same way... there are undoubtedly some key differences in how they behave, but it's not spelled out to the reader and I think the connections can be missed.
- The experiment/motivation I found most compelling was 4.1 (since it clearly matches the issue of partial observability) but we only see results compared to DQN... it feels like you don't put a compelling case for the non-Markovian benefits of ARM vs other policy gradient methods. Yes A3C and TRPO seem like they perform very poorly compared to ARM... but I'm left wondering how/why?
I feel like this paper is in a difficult position of trying to cover a lot of material/experiments in too short a paper.
A lot of the cited literature was also new to me, so it could be that I'm missing something about why this is so interesting.
However, I came away from this paper quite uncertain about the real benefits/differences of ARM versus other similar policy gradient methods... I also didn't feel the experimental evaluations drove a clear message except "ARM did better than all other methods on these experiments"... I'd want to understand how/why and whether we should expect this universally.
The focus on "regret minimization perspectives" didn't really get me too excited...
Overall I would vote against acceptance for this version.