title: Option-aware Temporally Abstracted Value for Offline Goal-Conditioned
    Reinforcement Learning
summary: Offline goal-conditioned reinforcement learning (GCRL) offers a practical
  learning paradigm where goal-reaching policies are trained from abundant
  unlabeled (reward-free) datasets without additional environment interaction.
  However, offline GCRL still struggles with long-horizon tasks, even with recent
  advances that employ hierarchical policy structures, such as HIQL. By
  identifying the root cause of this challenge, we observe the following
  insights: First, performance bottlenecks mainly stem from the high-level
  policy's inability to generate appropriate subgoals. Second, when learning the
  high-level policy in the long-horizon regime, the sign of the advantage signal
  frequently becomes incorrect. Thus, we argue that improving the value function
  to produce a clear advantage signal for learning the high-level policy is
  essential. In this paper, we propose a simple yet effective solution:
  Option-aware Temporally Abstracted value learning, dubbed OTA, which
  incorporates temporal abstraction into the temporal-difference learning
  process. By modifying the value update to be option-aware, the proposed
  learning scheme contracts the effective horizon length, enabling better
  advantage estimates even in long-horizon regimes. We experimentally show that
  the high-level policy extracted using the OTA value function achieves strong
  performance on complex tasks from OGBench, a recently proposed offline GCRL
  benchmark, including maze navigation and visual robotic manipulation
  environments.
paper_authors._id: 683572b271c57c48d732da5e
paper_authors.name: Hongjoon Ahn
paper_authors._id: 683572b271c57c48d732da5f
paper_authors.name: Heewoong Choi
paper_authors._id: 683572b271c57c48d732da60
paper_authors.name: Jisu Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T12:13:32.222Z
paper_authors._id: 644e78e8d6001776ed76bff0
paper_authors.avatarUrl: /avatars/04af3c60f0fd186723ec5691e7465ace.svg
paper_authors.fullname: Jisu Han
paper_authors.type: user
paper_authors.user: JisuHann
paper_authors._id: 683572b271c57c48d732da61
paper_authors.name: Taesup Moon
paper_title: Option-aware Temporally Abstracted Value for Offline Goal-Conditioned
    Reinforcement Learning
paper_summary: Offline goal-conditioned reinforcement learning (GCRL) offers a practical
  learning paradigm where goal-reaching policies are trained from abundant
  unlabeled (reward-free) datasets without additional environment interaction.
  However, offline GCRL still struggles with long-horizon tasks, even with recent
  advances that employ hierarchical policy structures, such as HIQL. By
  identifying the root cause of this challenge, we observe the following
  insights: First, performance bottlenecks mainly stem from the high-level
  policy's inability to generate appropriate subgoals. Second, when learning the
  high-level policy in the long-horizon regime, the sign of the advantage signal
  frequently becomes incorrect. Thus, we argue that improving the value function
  to produce a clear advantage signal for learning the high-level policy is
  essential. In this paper, we propose a simple yet effective solution:
  Option-aware Temporally Abstracted value learning, dubbed OTA, which
  incorporates temporal abstraction into the temporal-difference learning
  process. By modifying the value update to be option-aware, the proposed
  learning scheme contracts the effective horizon length, enabling better
  advantage estimates even in long-horizon regimes. We experimentally show that
  the high-level policy extracted using the OTA value function achieves strong
  performance on complex tasks from OGBench, a recently proposed offline GCRL
  benchmark, including maze navigation and visual robotic manipulation
  environments.
paper_ai_summary: Option-aware Temporally Abstracted (OTA) value learning improves offline goal-conditioned reinforcement learning performance by refining the high-level policy through better advantage estimates in long-horizon settings.
paper_ai_keywords: goal-conditioned reinforcement learning offline GCRL high-level policy subgoals advantage signal value function temporal abstraction temporal-difference learning OGBench maze navigation visual robotic manipulation