Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
  for Real-World Robot Policy Evaluation
Recent advancements in behavior cloning have enabled robots to perform
complex manipulation tasks. However, accurately assessing training performance
remains challenging, particularly for real-world applications, as behavior
cloning losses often correlate poorly with actual task success. Consequently,
researchers resort to success rate metrics derived from costly and
time-consuming real-world evaluations, making the identification of optimal
policies and detection of overfitting or underfitting impractical. To address
these issues, we propose real-is-sim, a novel behavior cloning framework that
incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
the entire policy development pipeline: data collection, training, and
deployment. By continuously aligning the simulated world with the physical
world, demonstrations can be collected in the real world with states extracted
from the simulator. The simulator enables flexible state representations by
rendering image inputs from any viewpoint or extracting low-level state
information from objects embodied within the scene. During training, policies
can be directly evaluated within the simulator in an offline and highly
parallelizable manner. Finally, during deployment, policies are run within the
simulator where the real robot directly tracks the simulated robot's joints,
effectively decoupling policy execution from real hardware and mitigating
traditional domain-transfer challenges. We validate real-is-sim on the PushT
manipulation task, demonstrating strong correlation between success rates
obtained in the simulator and real-world evaluations. Videos of our system can
be found at https://realissim.rai-inst.com.
67f3cea16faa90ba79c618d7
Jad Abou-Chakra
67f3cea16faa90ba79c618d8
Lingfeng Sun
claimed_verified
2025-04-15T08:01:01.320Z
666c66b4e4b8e059192831f9
https://cdn-avatars.huggingface.co/v1/production/uploads/666c66b4e4b8e059192831f9/I5VxIs7Q1L2RAn0Ov2IRc.jpeg
Lingfeng Sun
user
lsun-bdai
67f3cea16faa90ba79c618d9
Krishan Rana
67f3cea16faa90ba79c618da
Brandon May
claimed_verified
2025-04-08T06:52:36.290Z
666c76ad3ae28e1bcfe02c16
https://cdn-avatars.huggingface.co/v1/production/uploads/666c76ad3ae28e1bcfe02c16/GG9ImF8QF8BC__7ofbaxO.jpeg
Brandon B. May
user
bmay
67f3cea16faa90ba79c618db
Karl Schmeckpeper
67f3cea16faa90ba79c618dc
Maria Vittoria Minniti
67f3cea16faa90ba79c618dd
Laura Herlant
Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
  for Real-World Robot Policy Evaluation
Recent advancements in behavior cloning have enabled robots to perform
complex manipulation tasks. However, accurately assessing training performance
remains challenging, particularly for real-world applications, as behavior
cloning losses often correlate poorly with actual task success. Consequently,
researchers resort to success rate metrics derived from costly and
time-consuming real-world evaluations, making the identification of optimal
policies and detection of overfitting or underfitting impractical. To address
these issues, we propose real-is-sim, a novel behavior cloning framework that
incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
the entire policy development pipeline: data collection, training, and
deployment. By continuously aligning the simulated world with the physical
world, demonstrations can be collected in the real world with states extracted
from the simulator. The simulator enables flexible state representations by
rendering image inputs from any viewpoint or extracting low-level state
information from objects embodied within the scene. During training, policies
can be directly evaluated within the simulator in an offline and highly
parallelizable manner. Finally, during deployment, policies are run within the
simulator where the real robot directly tracks the simulated robot's joints,
effectively decoupling policy execution from real hardware and mitigating
traditional domain-transfer challenges. We validate real-is-sim on the PushT
manipulation task, demonstrating strong correlation between success rates
obtained in the simulator and real-world evaluations. Videos of our system can
be found at https://realissim.rai-inst.com.
A novel behavior cloning framework uses a dynamic digital twin for continuous alignment of simulated and real-world scenarios, overcoming assessment challenges and enabling efficient policy evaluation and deployment.
behavior cloning dynamic digital twin Embodied Gaussians policy development pipeline data collection offline evaluation parallelizable manner policy execution domain-transfer challenges PushT manipulation task