RECALL TRACES: BACKTRACKING MODELS FOR EFFICIENT REINFORCEMENT LEARNING
In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a backtracking model that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and samples which (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on-and off-policy RL algorithms across several environments and tasks.
Revision:
The authors have thoroughly addressed my review and I have consequently updated my rating accordingly.
Summary:
Model-free reinforcement learning is inefficient at exploration if rewards are
sparse / low probability.
The paper proposes a variational model for online learning to backtrack
state / action traces that lead to high reward states based on best previous
samples.
The backtracking models' generated recall traces are then used to augment policy
training by imitation learning, i.e. by optimizing policy to take actions that
are taken from the current states in generated recall traces.
Overall, the methodology seems akin to an adaptive importance sampling
approach for reinforcement learning.
Evaluation:
The paper gives a clear (at least mathematically) presentation of the core idea
but it some details about modeling choices seem to be missing.
The experimental evaluation seems preliminary and it is not fully evident when
and how the proposed method will be practically relevant (and not relevant).
My knowledgable of the previous literature is not sufficient to validate the
claimed novelty of the approach.
Details:
The paper is well written and easy to follow in general.
I'm not familiar enough with reinforcment learning benchmarks to judge the
quality of the experiments compared to the literature as a whole.
Although there are quite a few experiments they seem rather preliminary.
It is not clear whether enough work was done to understand the effect of the
many different hyperparameters that the proposed method surely must have.
The authors claim to show empirically that their method can improve sample
efficiency.
This is not necessarily a strong claim as such and could be achieved on
relatively simple tests.
In the discussion the authors claim their results indicate that their approach
is able to accelearte learning on a variety of tasks, also not a strong claim.
The paper could be improved by adding a more clear explanation of the exact way
by which the method helps with exploration and how it affects finding sparse
rewards (based on e.g. Figure 1).
It seems that since only knowledge of seen trajectories can be used to generate
paths to high reward states it only works for generating new trajectories
through previously visited states.
Questions that could be clarified:
- It is not entirely obvious to me what parametric models are used for the
backtracking distributions.
- Does this method not also potentially hinder exploration by making the agent
learn to go after the same high rewards / Does the direction of the variational
problem guarantee coverage of the support of the R > L distribution by samples?
- What would be the effect of a hyperparameter that balances learning the recall
traces and learning the true environment?
- Are there also reinforcement learning tasks where the proposed methods'
improvement is marginal and the extra modeling effort is not justified (e.g.
due to increase complexity).
Page 1: iwth (Typo)
Page 2: r(s_t) -> r(s_t, a_t)
Page 6: Prioritize d (Typo)