Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
<|reference_start|>Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model: We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023). Our analysis provides new theoretical results on categorical approaches to distributional RL, and also introduces a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. We also provide an experimental study comparing several model-based distributional RL algorithms, with several takeaways for practitioners.<|reference_end|>
@article{rowland2024near-minimax-optimal,
    title={Near-Minimax-Optimal Distributional Reinforcement Learning with a
  Generative Model},
    author={Mark Rowland, Li Kevin Wenliang, R'emi Munos, Clare Lyle, Yunhao
  Tang, Will Dabney},
    journal={arXiv preprint arXiv:2402.07598},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07598},
    primaryClass={cs.LG stat.ML}
}