SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
<|reference_start|>SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning: This paper studies the transfer reinforcement learning (RL) problem where multiple RL problems have different reward functions but share the same underlying transition dynamics. In this setting, the Q-function of each RL problem (task) can be decomposed into a successor feature (SF) and a reward mapping: the former characterizes the transition dynamics, and the latter characterizes the task-specific reward function. This Q-function decomposition, coupled with a policy improvement operator known as generalized policy improvement (GPI), reduces the sample complexity of finding the optimal Q-function, and thus the SF \& GPI framework exhibits promising empirical performance compared to traditional RL methods like Q-learning. However, its theoretical foundations remain largely unestablished, especially when learning the successor features using deep neural networks (SF-DQN). This paper studies the provable knowledge transfer using SFs-DQN in transfer RL problems. We establish the first convergence analysis with provable generalization guarantees for SF-DQN with GPI. The theory reveals that SF-DQN with GPI outperforms conventional RL approaches, such as deep Q-network, in terms of both faster convergence rate and better generalization. Numerical experiments on real and synthetic RL tasks support the superior performance of SF-DQN \& GPI, aligning with our theoretical findings.<|reference_end|>
@article{zhang2024sf-dqn:,
    title={SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep
  Reinforcement Learning},
    author={Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan,
  Songtao Lu, Pin-Yu Chen, Tianyi Chen, Meng Wang},
    journal={arXiv preprint arXiv:2405.15920},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15920},
    primaryClass={cs.LG stat.ML}
}