Optimal Transport-Assisted Risk-Sensitive Q-Learning
<|reference_start|>Optimal Transport-Assisted Risk-Sensitive Q-Learning: The primary goal of reinforcement learning is to develop decision-making policies that prioritize optimal performance without considering risk or safety. In contrast, safe reinforcement learning aims to mitigate or avoid unsafe states. This paper presents a risk-sensitive Q-learning algorithm that leverages optimal transport theory to enhance the agent safety. By integrating optimal transport into the Q-learning framework, our approach seeks to optimize the policy's expected return while minimizing the Wasserstein distance between the policy's stationary distribution and a predefined risk distribution, which encapsulates safety preferences from domain experts. We validate the proposed algorithm in a Gridworld environment. The results indicate that our method significantly reduces the frequency of visits to risky states and achieves faster convergence to a stable policy compared to the traditional Q-learning algorithm.<|reference_end|>
@article{shahrooei2024optimal,
    title={Optimal Transport-Assisted Risk-Sensitive Q-Learning},
    author={Zahra Shahrooei and Ali Baheri},
    journal={arXiv preprint arXiv:2406.11774},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11774},
    primaryClass={cs.LG cs.SY eess.SY}
}