Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
<|reference_start|>Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation: Distributional reinforcement learning improves performance by effectively capturing environmental stochasticity, but a comprehensive theoretical understanding of its effectiveness remains elusive. In this paper, we present a regret analysis for distributional reinforcement learning with general value function approximation in a finite episodic Markov decision process setting. We first introduce a key notion of Bellman unbiasedness for a tractable and exactly learnable update via statistical functional dynamic programming. Our theoretical results show that approximating the infinite-dimensional return distribution with a finite number of moment functionals is the only method to learn the statistical information unbiasedly, including nonlinear statistical functionals. Second, we propose a provably efficient algorithm, $\texttt{SF-LSVI}$, achieving a regret bound of $\tilde{O}(d_E H^{\frac{3}{2}}\sqrt{K})$ where $H$ is the horizon, $K$ is the number of episodes, and $d_E$ is the eluder dimension of a function class.<|reference_end|>
@article{cho2024tractable,
    title={Tractable and Provably Efficient Distributional Reinforcement Learning
  with General Value Function Approximation},
    author={Taehyun Cho, Seungyub Han, Kyungjae Lee, Seokhun Ju, Dohyeong Kim,
  Jungwoo Lee},
    journal={arXiv preprint arXiv:2407.21260},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.21260},
    primaryClass={cs.LG cs.AI stat.ML}
}