Value Improved Actor Critic Algorithms
<|reference_start|>Value Improved Actor Critic Algorithms: Many modern reinforcement learning algorithms build on the actor-critic (AC) framework: iterative improvement of a policy (the actor) using policy improvement operators and iterative approximation of the policy's value (the critic). In contrast, the popular value-based algorithm family employs improvement operators in the value update, to iteratively improve the value function directly. In this work, we propose a general extension to the AC framework that employs two separate improvement operators: one applied to the policy in the spirit of policy-based algorithms and one applied to the value in the spirit of value-based algorithms, which we dub Value-Improved AC (VI-AC). We design two practical VI-AC algorithms based in the popular online off-policy AC algorithms TD3 and DDPG. We evaluate VI-TD3 and VI-DDPG in the Mujoco benchmark and find that both improve upon or match the performance of their respective baselines in all environments tested.<|reference_end|>
@article{oren2024value,
    title={Value Improved Actor Critic Algorithms},
    author={Yaniv Oren, Moritz A. Zanger, Pascal R. van der Vaart, Matthijs T. J.
  Spaan, Wendelin Bohmer},
    journal={arXiv preprint arXiv:2406.01423},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01423},
    primaryClass={cs.LG cs.AI}
}