Dual Approximation Policy Optimization
<|reference_start|>Dual Approximation Policy Optimization: We propose Dual Approximation Policy Optimization (DAPO), a framework that incorporates general function approximation into policy mirror descent methods. In contrast to the popular approach of using the $L_2$-norm to measure function approximation errors, DAPO uses the dual Bregman divergence induced by the mirror map for policy projection. This duality framework has both theoretical and practical implications: not only does it achieve fast linear convergence with general function approximation, but it also includes several well-known practical methods as special cases, immediately providing strong convergence guarantees.<|reference_end|>
@article{xiong2024dual,
    title={Dual Approximation Policy Optimization},
    author={Zhihan Xiong, Maryam Fazel, Lin Xiao},
    journal={arXiv preprint arXiv:2410.01249},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01249},
    primaryClass={cs.LG}
}