title: Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or
    Robot Hardware
summary: Scaling robot learning requires vast and diverse datasets. Yet the prevailing
  data collection paradigm-human teleoperation-remains costly and constrained by
  manual effort and physical robot access. We introduce Real2Render2Real (R2R2R),
  a novel approach for generating robot training data without relying on object
  dynamics simulation or teleoperation of robot hardware. The input is a
  smartphone-captured scan of one or more objects and a single video of a human
  demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic
  demonstrations by reconstructing detailed 3D object geometry and appearance,
  and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to
  enable flexible asset generation and trajectory synthesis for both rigid and
  articulated objects, converting these representations to meshes to maintain
  compatibility with scalable rendering engines like IsaacLab but with collision
  modeling off. Robot demonstration data generated by R2R2R integrates directly
  with models that operate on robot proprioceptive states and image observations,
  such as vision-language-action models (VLA) and imitation learning policies.
  Physical experiments suggest that models trained on R2R2R data from a single
  human demonstration can match the performance of models trained on 150 human
  teleoperation demonstrations. Project page: https://real2render2real.com
paper_authors._id: 6827b0940908de1cafc267bb
paper_authors.name: Justin Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:46:53.727Z
paper_authors._id: 65d3e8ca15295f16a275443f
paper_authors.avatarUrl: /avatars/99740e6b4d663945ed24949ce482add5.svg
paper_authors.fullname: Justin Yu
paper_authors.type: user
paper_authors.user: uynitsuj
paper_authors._id: 6827b0940908de1cafc267bc
paper_authors.name: Letian Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:46:56.085Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 6827b0940908de1cafc267bd
paper_authors.name: Huang Huang
paper_authors._id: 6827b0940908de1cafc267be
paper_authors.name: Karim El-Refai
paper_authors._id: 6827b0940908de1cafc267bf
paper_authors.name: Rares Andrei Ambrus
paper_authors._id: 6827b0940908de1cafc267c0
paper_authors.name: Richard Cheng
paper_authors._id: 6827b0940908de1cafc267c1
paper_authors.name: Muhammad Zubair Irshad
paper_authors._id: 6827b0940908de1cafc267c2
paper_authors.name: Ken Goldberg
paper_title: Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or
    Robot Hardware
paper_summary: Scaling robot learning requires vast and diverse datasets. Yet the prevailing
  data collection paradigm-human teleoperation-remains costly and constrained by
  manual effort and physical robot access. We introduce Real2Render2Real (R2R2R),
  a novel approach for generating robot training data without relying on object
  dynamics simulation or teleoperation of robot hardware. The input is a
  smartphone-captured scan of one or more objects and a single video of a human
  demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic
  demonstrations by reconstructing detailed 3D object geometry and appearance,
  and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to
  enable flexible asset generation and trajectory synthesis for both rigid and
  articulated objects, converting these representations to meshes to maintain
  compatibility with scalable rendering engines like IsaacLab but with collision
  modeling off. Robot demonstration data generated by R2R2R integrates directly
  with models that operate on robot proprioceptive states and image observations,
  such as vision-language-action models (VLA) and imitation learning policies.
  Physical experiments suggest that models trained on R2R2R data from a single
  human demonstration can match the performance of models trained on 150 human
  teleoperation demonstrations. Project page: https://real2render2real.com
paper_ai_summary: R2R2R generates robot training data from 3D scans and videos, enabling high-fidelity demonstrations for various robot learning tasks.
paper_ai_keywords: 3D Gaussian Splatting trajectory synthesis 6-DoF vision-language-action models imitation learning policies proprioceptive states rendering engines collision modeling