SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting
<|reference_start|>SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting: Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data. Videos can be found on our project page: https://splatsim.github.io<|reference_end|>
@article{qureshi2024splatsim:,
    title={SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using
  Gaussian Splatting},
    author={Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held,
  George Kantor, Abhisesh Silwal},
    journal={arXiv preprint arXiv:2409.10161},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.10161},
    primaryClass={cs.RO cs.AI cs.CV cs.LG}
}