Transferable Reinforcement Learning via Generalized Occupancy Models
<|reference_start|>Transferable Reinforcement Learning via Generalized Occupancy Models: Intelligent agents must be generalists, capable of quickly adapting to various tasks. In reinforcement learning (RL), model-based RL learns a dynamics model of the world, in principle enabling transfer to arbitrary reward functions through planning. However, autoregressive model rollouts suffer from compounding error, making model-based RL ineffective for long-horizon problems. Successor features offer an alternative by modeling a policy's long-term state occupancy, reducing policy evaluation under new tasks to linear reward regression. Yet, policy improvement with successor features can be challenging. This work proposes a novel class of models, i.e., generalized occupancy models (GOMs), that learn a distribution of successor features from a stationary dataset, along with a policy that acts to realize different successor features. These models can quickly select the optimal action for arbitrary new tasks. By directly modeling long-term outcomes in the dataset, GOMs avoid compounding error while enabling rapid transfer across reward functions. We present a practical instantiation of GOMs using diffusion models and show their efficacy as a new class of transferable models, both theoretically and empirically across various simulated robotics problems. Videos and code at https://weirdlabuw.github.io/gom/.<|reference_end|>
@article{zhu2024distributional,
    title={Distributional Successor Features Enable Zero-Shot Policy Optimization},
    author={Chuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du, Abhishek Gupta},
    journal={arXiv preprint arXiv:2403.06328},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.06328},
    primaryClass={cs.LG}
}