Risk-Averse Reinforcement Learning with Itakura-Saito Loss
Risk-averse reinforcement learning finds application in various high-stakes
fields. Unlike classical reinforcement learning, which aims to maximize
expected returns, risk-averse agents choose policies that minimize risk,
occasionally sacrificing expected value. These preferences can be framed
through utility theory. We focus on the specific case of the exponential
utility function, where we can derive the Bellman equations and employ various
reinforcement learning algorithms with few modifications. However, these
methods suffer from numerical instability due to the need for exponent
computation throughout the process. To address this, we introduce a numerically
stable and mathematically sound loss function based on the Itakura-Saito
divergence for learning state-value and action-value functions. We evaluate our
proposed loss function against established alternatives, both theoretically and
empirically. In the experimental section, we explore multiple financial
scenarios, some with known analytical solutions, and show that our loss
function outperforms the alternatives.
68302d01b85e3ed6a61e6476
Igor Udovichenko
extracted_confirmed
2025-05-23T08:16:34.263Z
67a33f0e36fccbd55d6e8f7f
https://cdn-avatars.huggingface.co/v1/production/uploads/67a33f0e36fccbd55d6e8f7f/o_2UU5bxtnPmPdKo-jg7x.png
Igor Udovichenko
user
i-udovichenko
68302d01b85e3ed6a61e6477
Olivier Croissant
68302d01b85e3ed6a61e6478
Anita Toleutaeva
68302d01b85e3ed6a61e6479
Evgeny Burnaev
68302d01b85e3ed6a61e647a
Alexander Korotin
Risk-Averse Reinforcement Learning with Itakura-Saito Loss
Risk-averse reinforcement learning finds application in various high-stakes
fields. Unlike classical reinforcement learning, which aims to maximize
expected returns, risk-averse agents choose policies that minimize risk,
occasionally sacrificing expected value. These preferences can be framed
through utility theory. We focus on the specific case of the exponential
utility function, where we can derive the Bellman equations and employ various
reinforcement learning algorithms with few modifications. However, these
methods suffer from numerical instability due to the need for exponent
computation throughout the process. To address this, we introduce a numerically
stable and mathematically sound loss function based on the Itakura-Saito
divergence for learning state-value and action-value functions. We evaluate our
proposed loss function against established alternatives, both theoretically and
empirically. In the experimental section, we explore multiple financial
scenarios, some with known analytical solutions, and show that our loss
function outperforms the alternatives.
Proposed Itakura-Saito divergence-based loss function enhances numerical stability in risk-averse reinforcement learning using exponential utility functions.
reinforcement learning risk-averse utility theory exponential utility function Bellman equations numerical instability Itakura-Saito divergence state-value functions action-value functions