title: Risk-Averse Reinforcement Learning with Itakura-Saito Loss
summary: Risk-averse reinforcement learning finds application in various high-stakes
  fields. Unlike classical reinforcement learning, which aims to maximize
  expected returns, risk-averse agents choose policies that minimize risk,
  occasionally sacrificing expected value. These preferences can be framed
  through utility theory. We focus on the specific case of the exponential
  utility function, where we can derive the Bellman equations and employ various
  reinforcement learning algorithms with few modifications. However, these
  methods suffer from numerical instability due to the need for exponent
  computation throughout the process. To address this, we introduce a numerically
  stable and mathematically sound loss function based on the Itakura-Saito
  divergence for learning state-value and action-value functions. We evaluate our
  proposed loss function against established alternatives, both theoretically and
  empirically. In the experimental section, we explore multiple financial
  scenarios, some with known analytical solutions, and show that our loss
  function outperforms the alternatives.
paper_authors._id: 68302d01b85e3ed6a61e6476
paper_authors.name: Igor Udovichenko
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-23T08:16:34.263Z
paper_authors._id: 67a33f0e36fccbd55d6e8f7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67a33f0e36fccbd55d6e8f7f/o_2UU5bxtnPmPdKo-jg7x.png
paper_authors.fullname: Igor Udovichenko
paper_authors.type: user
paper_authors.user: i-udovichenko
paper_authors._id: 68302d01b85e3ed6a61e6477
paper_authors.name: Olivier Croissant
paper_authors._id: 68302d01b85e3ed6a61e6478
paper_authors.name: Anita Toleutaeva
paper_authors._id: 68302d01b85e3ed6a61e6479
paper_authors.name: Evgeny Burnaev
paper_authors._id: 68302d01b85e3ed6a61e647a
paper_authors.name: Alexander Korotin
paper_title: Risk-Averse Reinforcement Learning with Itakura-Saito Loss
paper_summary: Risk-averse reinforcement learning finds application in various high-stakes
  fields. Unlike classical reinforcement learning, which aims to maximize
  expected returns, risk-averse agents choose policies that minimize risk,
  occasionally sacrificing expected value. These preferences can be framed
  through utility theory. We focus on the specific case of the exponential
  utility function, where we can derive the Bellman equations and employ various
  reinforcement learning algorithms with few modifications. However, these
  methods suffer from numerical instability due to the need for exponent
  computation throughout the process. To address this, we introduce a numerically
  stable and mathematically sound loss function based on the Itakura-Saito
  divergence for learning state-value and action-value functions. We evaluate our
  proposed loss function against established alternatives, both theoretically and
  empirically. In the experimental section, we explore multiple financial
  scenarios, some with known analytical solutions, and show that our loss
  function outperforms the alternatives.
paper_ai_summary: Proposed Itakura-Saito divergence-based loss function enhances numerical stability in risk-averse reinforcement learning using exponential utility functions.
paper_ai_keywords: reinforcement learning risk-averse utility theory exponential utility function Bellman equations numerical instability Itakura-Saito divergence state-value functions action-value functions