title: Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
    for Real-World Robot Policy Evaluation
summary: Recent advancements in behavior cloning have enabled robots to perform
  complex manipulation tasks. However, accurately assessing training performance
  remains challenging, particularly for real-world applications, as behavior
  cloning losses often correlate poorly with actual task success. Consequently,
  researchers resort to success rate metrics derived from costly and
  time-consuming real-world evaluations, making the identification of optimal
  policies and detection of overfitting or underfitting impractical. To address
  these issues, we propose real-is-sim, a novel behavior cloning framework that
  incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
  the entire policy development pipeline: data collection, training, and
  deployment. By continuously aligning the simulated world with the physical
  world, demonstrations can be collected in the real world with states extracted
  from the simulator. The simulator enables flexible state representations by
  rendering image inputs from any viewpoint or extracting low-level state
  information from objects embodied within the scene. During training, policies
  can be directly evaluated within the simulator in an offline and highly
  parallelizable manner. Finally, during deployment, policies are run within the
  simulator where the real robot directly tracks the simulated robot's joints,
  effectively decoupling policy execution from real hardware and mitigating
  traditional domain-transfer challenges. We validate real-is-sim on the PushT
  manipulation task, demonstrating strong correlation between success rates
  obtained in the simulator and real-world evaluations. Videos of our system can
  be found at https://realissim.rai-inst.com.
paper_authors._id: 67f3cea16faa90ba79c618d7
paper_authors.name: Jad Abou-Chakra
paper_authors._id: 67f3cea16faa90ba79c618d8
paper_authors.name: Lingfeng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T08:01:01.320Z
paper_authors._id: 666c66b4e4b8e059192831f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666c66b4e4b8e059192831f9/I5VxIs7Q1L2RAn0Ov2IRc.jpeg
paper_authors.fullname: Lingfeng Sun
paper_authors.type: user
paper_authors.user: lsun-bdai
paper_authors._id: 67f3cea16faa90ba79c618d9
paper_authors.name: Krishan Rana
paper_authors._id: 67f3cea16faa90ba79c618da
paper_authors.name: Brandon May
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:52:36.290Z
paper_authors._id: 666c76ad3ae28e1bcfe02c16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666c76ad3ae28e1bcfe02c16/GG9ImF8QF8BC__7ofbaxO.jpeg
paper_authors.fullname: Brandon B. May
paper_authors.type: user
paper_authors.user: bmay
paper_authors._id: 67f3cea16faa90ba79c618db
paper_authors.name: Karl Schmeckpeper
paper_authors._id: 67f3cea16faa90ba79c618dc
paper_authors.name: Maria Vittoria Minniti
paper_authors._id: 67f3cea16faa90ba79c618dd
paper_authors.name: Laura Herlant
paper_title: Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
    for Real-World Robot Policy Evaluation
paper_summary: Recent advancements in behavior cloning have enabled robots to perform
  complex manipulation tasks. However, accurately assessing training performance
  remains challenging, particularly for real-world applications, as behavior
  cloning losses often correlate poorly with actual task success. Consequently,
  researchers resort to success rate metrics derived from costly and
  time-consuming real-world evaluations, making the identification of optimal
  policies and detection of overfitting or underfitting impractical. To address
  these issues, we propose real-is-sim, a novel behavior cloning framework that
  incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
  the entire policy development pipeline: data collection, training, and
  deployment. By continuously aligning the simulated world with the physical
  world, demonstrations can be collected in the real world with states extracted
  from the simulator. The simulator enables flexible state representations by
  rendering image inputs from any viewpoint or extracting low-level state
  information from objects embodied within the scene. During training, policies
  can be directly evaluated within the simulator in an offline and highly
  parallelizable manner. Finally, during deployment, policies are run within the
  simulator where the real robot directly tracks the simulated robot's joints,
  effectively decoupling policy execution from real hardware and mitigating
  traditional domain-transfer challenges. We validate real-is-sim on the PushT
  manipulation task, demonstrating strong correlation between success rates
  obtained in the simulator and real-world evaluations. Videos of our system can
  be found at https://realissim.rai-inst.com.
paper_ai_summary: A novel behavior cloning framework uses a dynamic digital twin for continuous alignment of simulated and real-world scenarios, overcoming assessment challenges and enabling efficient policy evaluation and deployment.
paper_ai_keywords: behavior cloning dynamic digital twin Embodied Gaussians policy development pipeline data collection offline evaluation parallelizable manner policy execution domain-transfer challenges PushT manipulation task