title: Open-World Skill Discovery from Unsegmented Demonstrations
summary: Learning skills in open-world environments is essential for developing agents
  capable of handling a variety of tasks by combining basic skills. Online
  demonstration videos are typically long but unsegmented, making them difficult
  to segment and label with skill identifiers. Unlike existing methods that rely
  on sequence sampling or human labeling, we have developed a self-supervised
  learning-based approach to segment these long videos into a series of
  semantic-aware and skill-consistent segments. Drawing inspiration from human
  cognitive event segmentation theory, we introduce Skill Boundary Detection
  (SBD), an annotation-free temporal video segmentation algorithm. SBD detects
  skill boundaries in a video by leveraging prediction errors from a pretrained
  unconditional action-prediction model. This approach is based on the assumption
  that a significant increase in prediction error indicates a shift in the skill
  being executed. We evaluated our method in Minecraft, a rich open-world
  simulator with extensive gameplay videos available online. Our SBD-generated
  segments improved the average performance of conditioned policies by 63.7% and
  52.1% on short-term atomic skill tasks, and their corresponding hierarchical
  agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the
  diverse YouTube videos to train instruction-following agents. The project page
  can be found in https://craftjarvis.github.io/SkillDiscovery.
paper_authors._id: 67d8030568c586e2c8fcac8f
paper_authors.name: Jingwen Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:42:10.438Z
paper_authors._id: 66962a99a51ccecc90a662ae
paper_authors.avatarUrl: /avatars/b4e200cc42eef5990678c168e6e51ab2.svg
paper_authors.fullname: DJW
paper_authors.type: user
paper_authors.user: fatty-belly
paper_authors._id: 67d8030568c586e2c8fcac90
paper_authors.name: Zihao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:42:12.462Z
paper_authors._id: 642e8c99c1b0f8e4e76bcaab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: zhwang4ai
paper_authors._id: 67d8030568c586e2c8fcac91
paper_authors.name: Shaofei Cai
paper_authors._id: 67d8030568c586e2c8fcac92
paper_authors.name: Anji Liu
paper_authors._id: 67d8030568c586e2c8fcac93
paper_authors.name: Yitao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:38:16.563Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: Open-World Skill Discovery from Unsegmented Demonstrations
paper_summary: Learning skills in open-world environments is essential for developing agents
  capable of handling a variety of tasks by combining basic skills. Online
  demonstration videos are typically long but unsegmented, making them difficult
  to segment and label with skill identifiers. Unlike existing methods that rely
  on sequence sampling or human labeling, we have developed a self-supervised
  learning-based approach to segment these long videos into a series of
  semantic-aware and skill-consistent segments. Drawing inspiration from human
  cognitive event segmentation theory, we introduce Skill Boundary Detection
  (SBD), an annotation-free temporal video segmentation algorithm. SBD detects
  skill boundaries in a video by leveraging prediction errors from a pretrained
  unconditional action-prediction model. This approach is based on the assumption
  that a significant increase in prediction error indicates a shift in the skill
  being executed. We evaluated our method in Minecraft, a rich open-world
  simulator with extensive gameplay videos available online. Our SBD-generated
  segments improved the average performance of conditioned policies by 63.7% and
  52.1% on short-term atomic skill tasks, and their corresponding hierarchical
  agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the
  diverse YouTube videos to train instruction-following agents. The project page
  can be found in https://craftjarvis.github.io/SkillDiscovery.
paper_ai_summary: A self-supervised Skill Boundary Detection method segments long, unlabelled demonstration videos into skill-consistent segments, enhancing performance in open-world simulations.
paper_ai_keywords: Skill Boundary Detection self-supervised learning temporal video segmentation prediction errors action-prediction model open-world environments Minecraft atomic skill tasks long-horizon tasks instruction-following agents