Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or
  Robot Hardware
Scaling robot learning requires vast and diverse datasets. Yet the prevailing
data collection paradigm-human teleoperation-remains costly and constrained by
manual effort and physical robot access. We introduce Real2Render2Real (R2R2R),
a novel approach for generating robot training data without relying on object
dynamics simulation or teleoperation of robot hardware. The input is a
smartphone-captured scan of one or more objects and a single video of a human
demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic
demonstrations by reconstructing detailed 3D object geometry and appearance,
and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to
enable flexible asset generation and trajectory synthesis for both rigid and
articulated objects, converting these representations to meshes to maintain
compatibility with scalable rendering engines like IsaacLab but with collision
modeling off. Robot demonstration data generated by R2R2R integrates directly
with models that operate on robot proprioceptive states and image observations,
such as vision-language-action models (VLA) and imitation learning policies.
Physical experiments suggest that models trained on R2R2R data from a single
human demonstration can match the performance of models trained on 150 human
teleoperation demonstrations. Project page: https://real2render2real.com
6827b0940908de1cafc267bb
Justin Yu
claimed_verified
2025-05-18T08:46:53.727Z
65d3e8ca15295f16a275443f
/avatars/99740e6b4d663945ed24949ce482add5.svg
Justin Yu
user
uynitsuj
6827b0940908de1cafc267bc
Letian Fu
claimed_verified
2025-05-18T08:46:56.085Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
6827b0940908de1cafc267bd
Huang Huang
6827b0940908de1cafc267be
Karim El-Refai
6827b0940908de1cafc267bf
Rares Andrei Ambrus
6827b0940908de1cafc267c0
Richard Cheng
6827b0940908de1cafc267c1
Muhammad Zubair Irshad
6827b0940908de1cafc267c2
Ken Goldberg
Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or
  Robot Hardware
Scaling robot learning requires vast and diverse datasets. Yet the prevailing
data collection paradigm-human teleoperation-remains costly and constrained by
manual effort and physical robot access. We introduce Real2Render2Real (R2R2R),
a novel approach for generating robot training data without relying on object
dynamics simulation or teleoperation of robot hardware. The input is a
smartphone-captured scan of one or more objects and a single video of a human
demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic
demonstrations by reconstructing detailed 3D object geometry and appearance,
and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to
enable flexible asset generation and trajectory synthesis for both rigid and
articulated objects, converting these representations to meshes to maintain
compatibility with scalable rendering engines like IsaacLab but with collision
modeling off. Robot demonstration data generated by R2R2R integrates directly
with models that operate on robot proprioceptive states and image observations,
such as vision-language-action models (VLA) and imitation learning policies.
Physical experiments suggest that models trained on R2R2R data from a single
human demonstration can match the performance of models trained on 150 human
teleoperation demonstrations. Project page: https://real2render2real.com
R2R2R generates robot training data from 3D scans and videos, enabling high-fidelity demonstrations for various robot learning tasks.
3D Gaussian Splatting trajectory synthesis 6-DoF vision-language-action models imitation learning policies proprioceptive states rendering engines collision modeling