Biological intelligence can learn to solve many diverse tasks in a data efficient manner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired without explicit supervision in an intrinsically driven fashion. This is in contrast to the state-of-the-art reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn a basis set of policies, which, when recombined through generalised policy improvement, come with guarantees on the coverage of the final task space. In particular, we concentrate on solving goal-based downstream tasks where the execution order of actions is not important. We demonstrate both theoretically and empirically that learning a small number of policies that reach intrinsically specified goal regions in a disentangled latent space can be re-used to quickly achieve a high level of performance on an exponentially larger number of externally specified, often significantly more complex downstream tasks. Our learning pipeline consists of two stages. First, the agent learns to perform intrinsically generated, goal-based tasks in the total absence of environmental rewards. Second, the agent leverages this experience to quickly achieve a high level of performance on numerous diverse externally specified tasks.
This paper proposes to pre-train policies on some goal-reaching tasks, and then leverage the associated successor features to improve the learning of a new task. The method heavily draws from the Generalized Policy Evaluation/Improvement framework without adding much to it. The only relevant point would be showing (as the title indicates) how to obtain disentangled cumulants, and whether they help transfer to new tasks. Nevertheless, both the definition, the full method, and the claimed benefits are quite ambiguous.
Among other concerns showing that the theory needs more formal treatment, the pillar definition of “Optimal independent controllability” is very confusing because it seems to depend on “a trajectory generated by following \pi_i^*”. But what If the environment is stochastic? Then following that policy might give different trajectories! This definition needs to be revisited. More concerning examples are given at the end of this review.
On the experimental side, Fig. 4 is the only reported result, and it has an x-axis that is not clearly explained. What are the “steps (min)”?
It is also not clear what they mean by the “off-diagonal trick”, which seems so important for the good performance.
Furthermore, it seems that their method doesn’t really learn anything new in most of the tasks, it just stays at the same performance that is started with after the whole pre-training steps. It is not clearly stated how much computation effort is required to obtain the desired cumulants, and this invalidates quite strongly any result they report. Even if there’s no “reward” needed during the pre-training, which arguably is not even true because you do need the rewards related to whether you have achieved a specific change in a feature!
In fact, it would be greatly appreciated if the “final” tasks could be expressed in a similar notation than the rest of the pre-training tasks, or vice-versa. As far as I understand, the pre-training tasks consist of making a certain feature fall into a certain subset of its possible values. Can’t the final tasks, like “move the agent to the top right” be also expressed in that form. The link between the two kinds of tasks needs to be much more explicit to be able to assess the relevance of this work.
Finally, they only test their algorithm on Spritworld, which is a small discrete state-action space environment. Even if they try different kinds of tasks in this environment, more detailed analysis or more extensive experiments are needed to assess the benefits of the proposed approach.
This is particularly timely because their method relies on a discretization of some given features that represent the state, which will probably not be very practical in higher dimensional environments.
Finally, I would like a comment on how this method interacts with discrete versus continuous action-state spaces.
Misc comments:
- Why do the authors introduce the terminology “Endogenous RL”, and then say it’s the same as doing RL with intrinsic motivation? This seems like introducing a new name for the same concept, which seems pointless and confusing.
- The connection with “latent learning” of Tolman 1984 is very unclear.
- There’s a “Representation Learning” section, but it’s not clear at all whether any features are actually learned, or whether the features are actually hand-defined. Is the number of features n also hand-defined?
- There might be a typo in the first sentence after equation (9): “While \phi_w is not guaranteed to be optimal with respect to \phi_w”.
Because of all these concerns, I suggest the paper to be weakly rejected.