title: X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
summary: Human videos offer a scalable way to train robot manipulation policies, but
  lack the action labels needed by standard imitation learning algorithms.
  Existing cross-embodiment approaches try to map human motion to robot actions,
  but often fail when the embodiments differ significantly. We propose X-Sim, a
  real-to-sim-to-real framework that uses object motion as a dense and
  transferable signal for learning robot policies. X-Sim starts by reconstructing
  a photorealistic simulation from an RGBD human video and tracking object
  trajectories to define object-centric rewards. These rewards are used to train
  a reinforcement learning (RL) policy in simulation. The learned policy is then
  distilled into an image-conditioned diffusion policy using synthetic rollouts
  rendered with varied viewpoints and lighting. To transfer to the real world,
  X-Sim introduces an online domain adaptation technique that aligns real and
  simulated observations during deployment. Importantly, X-Sim does not require
  any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2
  environments and show that it: (1) improves task progress by 30% on average
  over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with
  10x less data collection time, and (3) generalizes to new camera viewpoints and
  test-time changes. Code and videos are available at
  https://portal-cornell.github.io/X-Sim/.
paper_authors._id: 68276829c73b5a259249b459
paper_authors.name: Prithwish Dan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T07:26:23.583Z
paper_authors._id: 65e8c6cae214f37d85e341c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e8c6cae214f37d85e341c0/CtpYeBEqap9TwomncZI8V.png
paper_authors.fullname: Prithwish Dan
paper_authors.type: user
paper_authors.user: prithwishdan
paper_authors._id: 68276829c73b5a259249b45a
paper_authors.name: Kushal Kedia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T07:26:38.361Z
paper_authors._id: 65e1ff68ec41b21de6201cbd
paper_authors.avatarUrl: /avatars/498f8d0b5d83c18e9aefdf730f5be170.svg
paper_authors.fullname: Kushal Kedia
paper_authors.type: user
paper_authors.user: Kushal20
paper_authors._id: 68276829c73b5a259249b45b
paper_authors.name: Angela Chao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T07:26:45.371Z
paper_authors._id: 65e3d60d00680b19bc015b0a
paper_authors.avatarUrl: /avatars/f0d8e56a8181f0a59439ebe961ef5b06.svg
paper_authors.fullname: Angela Chao
paper_authors.type: user
paper_authors.user: ac2323
paper_authors._id: 68276829c73b5a259249b45c
paper_authors.name: Edward Weiyi Duan
paper_authors._id: 68276829c73b5a259249b45d
paper_authors.name: Maximus Adrian Pace
paper_authors._id: 68276829c73b5a259249b45e
paper_authors.name: Wei-Chiu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T07:27:01.583Z
paper_authors._id: 624fb0c8841b0a7f1b583e72
paper_authors.avatarUrl: /avatars/a695a528817f57a98c946056d6f9b2f9.svg
paper_authors.fullname: Wei-Chiu Ma
paper_authors.type: user
paper_authors.user: weichium
paper_authors._id: 68276829c73b5a259249b45f
paper_authors.name: Sanjiban Choudhury
paper_title: X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
paper_summary: Human videos offer a scalable way to train robot manipulation policies, but
  lack the action labels needed by standard imitation learning algorithms.
  Existing cross-embodiment approaches try to map human motion to robot actions,
  but often fail when the embodiments differ significantly. We propose X-Sim, a
  real-to-sim-to-real framework that uses object motion as a dense and
  transferable signal for learning robot policies. X-Sim starts by reconstructing
  a photorealistic simulation from an RGBD human video and tracking object
  trajectories to define object-centric rewards. These rewards are used to train
  a reinforcement learning (RL) policy in simulation. The learned policy is then
  distilled into an image-conditioned diffusion policy using synthetic rollouts
  rendered with varied viewpoints and lighting. To transfer to the real world,
  X-Sim introduces an online domain adaptation technique that aligns real and
  simulated observations during deployment. Importantly, X-Sim does not require
  any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2
  environments and show that it: (1) improves task progress by 30% on average
  over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with
  10x less data collection time, and (3) generalizes to new camera viewpoints and
  test-time changes. Code and videos are available at
  https://portal-cornell.github.io/X-Sim/.
paper_ai_summary: X-Sim uses photorealistic simulations and reinforcement learning to train robot manipulation policies from human videos without labeled actions, enhancing generalization and efficiency.
paper_ai_keywords: photorealistic simulation object-centric rewards reinforcement learning diffusion policy synthetic rollouts domain adaptation hand-tracking sim-to-real