Off-policy Distributional Q($\lambda$): Distributional RL without Importance Sampling
<|reference_start|>Off-policy Distributional Q($\lambda$): Distributional RL without Importance Sampling: We introduce off-policy distributional Q($\lambda$), a new addition to the family of off-policy distributional evaluation algorithms. Off-policy distributional Q($\lambda$) does not apply importance sampling for off-policy learning, which introduces intriguing interactions with signed measures. Such unique properties distributional Q($\lambda$) from other existing alternatives such as distributional Retrace. We characterize the algorithmic properties of distributional Q($\lambda$) and validate theoretical insights with tabular experiments. We show how distributional Q($\lambda$)-C51, a combination of Q($\lambda$) with the C51 agent, exhibits promising results on deep RL benchmarks.<|reference_end|>
@article{tang2024off-policy,
    title={Off-policy Distributional Q($\lambda$): Distributional RL without
  Importance Sampling},
    author={Yunhao Tang, Mark Rowland, R'emi Munos, Bernardo 'Avila Pires, Will
  Dabney},
    journal={arXiv preprint arXiv:2402.05766},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.05766},
    primaryClass={cs.LG}
}