Option-aware Temporally Abstracted Value for Offline Goal-Conditioned
  Reinforcement Learning
Offline goal-conditioned reinforcement learning (GCRL) offers a practical
learning paradigm where goal-reaching policies are trained from abundant
unlabeled (reward-free) datasets without additional environment interaction.
However, offline GCRL still struggles with long-horizon tasks, even with recent
advances that employ hierarchical policy structures, such as HIQL. By
identifying the root cause of this challenge, we observe the following
insights: First, performance bottlenecks mainly stem from the high-level
policy's inability to generate appropriate subgoals. Second, when learning the
high-level policy in the long-horizon regime, the sign of the advantage signal
frequently becomes incorrect. Thus, we argue that improving the value function
to produce a clear advantage signal for learning the high-level policy is
essential. In this paper, we propose a simple yet effective solution:
Option-aware Temporally Abstracted value learning, dubbed OTA, which
incorporates temporal abstraction into the temporal-difference learning
process. By modifying the value update to be option-aware, the proposed
learning scheme contracts the effective horizon length, enabling better
advantage estimates even in long-horizon regimes. We experimentally show that
the high-level policy extracted using the OTA value function achieves strong
performance on complex tasks from OGBench, a recently proposed offline GCRL
benchmark, including maze navigation and visual robotic manipulation
environments.
683572b271c57c48d732da5e
Hongjoon Ahn
683572b271c57c48d732da5f
Heewoong Choi
683572b271c57c48d732da60
Jisu Han
claimed_verified
2025-05-27T12:13:32.222Z
644e78e8d6001776ed76bff0
/avatars/04af3c60f0fd186723ec5691e7465ace.svg
Jisu Han
user
JisuHann
683572b271c57c48d732da61
Taesup Moon
Option-aware Temporally Abstracted Value for Offline Goal-Conditioned
  Reinforcement Learning
Offline goal-conditioned reinforcement learning (GCRL) offers a practical
learning paradigm where goal-reaching policies are trained from abundant
unlabeled (reward-free) datasets without additional environment interaction.
However, offline GCRL still struggles with long-horizon tasks, even with recent
advances that employ hierarchical policy structures, such as HIQL. By
identifying the root cause of this challenge, we observe the following
insights: First, performance bottlenecks mainly stem from the high-level
policy's inability to generate appropriate subgoals. Second, when learning the
high-level policy in the long-horizon regime, the sign of the advantage signal
frequently becomes incorrect. Thus, we argue that improving the value function
to produce a clear advantage signal for learning the high-level policy is
essential. In this paper, we propose a simple yet effective solution:
Option-aware Temporally Abstracted value learning, dubbed OTA, which
incorporates temporal abstraction into the temporal-difference learning
process. By modifying the value update to be option-aware, the proposed
learning scheme contracts the effective horizon length, enabling better
advantage estimates even in long-horizon regimes. We experimentally show that
the high-level policy extracted using the OTA value function achieves strong
performance on complex tasks from OGBench, a recently proposed offline GCRL
benchmark, including maze navigation and visual robotic manipulation
environments.
Option-aware Temporally Abstracted (OTA) value learning improves offline goal-conditioned reinforcement learning performance by refining the high-level policy through better advantage estimates in long-horizon settings.
goal-conditioned reinforcement learning offline GCRL high-level policy subgoals advantage signal value function temporal abstraction temporal-difference learning OGBench maze navigation visual robotic manipulation