Reinforcement Learning for Adaptive MCMC
<|reference_start|>Reinforcement Learning for Adaptive MCMC: An informal observation, made by several authors, is that the adaptive design of a Markov transition kernel has the flavour of a reinforcement learning task. Yet, to-date it has remained unclear how to actually exploit modern reinforcement learning technologies for adaptive MCMC. The aim of this paper is to set out a general framework, called Reinforcement Learning Metropolis--Hastings, that is theoretically supported and empirically validated. Our principal focus is on learning fast-mixing Metropolis--Hastings transition kernels, which we cast as deterministic policies and optimise via a policy gradient. Control of the learning rate provably ensures conditions for ergodicity are satisfied. The methodology is used to construct a gradient-free sampler that out-performs a popular gradient-free adaptive Metropolis--Hastings algorithm on $\approx 90 \%$ of tasks in the PosteriorDB benchmark.<|reference_end|>
@article{wang2024reinforcement,
    title={Reinforcement Learning for Adaptive MCMC},
    author={Congye Wang, Wilson Chen, Heishiro Kanagawa, Chris. J. Oates},
    journal={arXiv preprint arXiv:2405.13574},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13574},
    primaryClass={stat.CO cs.LG}
}