title: CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
summary: Recovering high-quality 3D scenes from a single RGB image is a challenging
  task in computer graphics. Current methods often struggle with domain-specific
  limitations or low-quality object generation. To address these, we propose CAST
  (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel
  method for 3D scene reconstruction and recovery. CAST starts by extracting
  object-level 2D segmentation and relative depth information from the input
  image, followed by using a GPT-based model to analyze inter-object spatial
  relationships. This enables the understanding of how objects relate to each
  other within the scene, ensuring more coherent reconstruction. CAST then
  employs an occlusion-aware large-scale 3D generation model to independently
  generate each object's full geometry, using MAE and point cloud conditioning to
  mitigate the effects of occlusions and partial object information, ensuring
  accurate alignment with the source image's geometry and texture. To align each
  object with the scene, the alignment generation model computes the necessary
  transformations, allowing the generated meshes to be accurately placed and
  integrated into the scene's point cloud. Finally, CAST incorporates a
  physics-aware correction step that leverages a fine-grained relation graph to
  generate a constraint graph. This graph guides the optimization of object
  poses, ensuring physical consistency and spatial coherence. By utilizing Signed
  Distance Fields (SDF), the model effectively addresses issues such as
  occlusions, object penetration, and floating objects, ensuring that the
  generated scene accurately reflects real-world physical interactions. CAST can
  be leveraged in robotics, enabling efficient real-to-simulation workflows and
  providing realistic, scalable simulation environments for robotic systems.
paper_authors._id: 67b60a5c92649eb787234701
paper_authors.name: Kaixin Yao
paper_authors._id: 67b60a5c92649eb787234702
paper_authors.name: Longwen Zhang
paper_authors._id: 67b60a5c92649eb787234703
paper_authors.name: Xinhao Yan
paper_authors._id: 67b60a5c92649eb787234704
paper_authors.name: Yan Zeng
paper_authors._id: 67b60a5c92649eb787234705
paper_authors.name: Qixuan Zhang
paper_authors._id: 67b60a5c92649eb787234706
paper_authors.name: Lan Xu
paper_authors._id: 67b60a5c92649eb787234707
paper_authors.name: Wei Yang
paper_authors._id: 67b60a5c92649eb787234708
paper_authors.name: Jiayuan Gu
paper_authors._id: 67b60a5c92649eb787234709
paper_authors.name: Jingyi Yu
paper_title: CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
paper_summary: Recovering high-quality 3D scenes from a single RGB image is a challenging
  task in computer graphics. Current methods often struggle with domain-specific
  limitations or low-quality object generation. To address these, we propose CAST
  (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel
  method for 3D scene reconstruction and recovery. CAST starts by extracting
  object-level 2D segmentation and relative depth information from the input
  image, followed by using a GPT-based model to analyze inter-object spatial
  relationships. This enables the understanding of how objects relate to each
  other within the scene, ensuring more coherent reconstruction. CAST then
  employs an occlusion-aware large-scale 3D generation model to independently
  generate each object's full geometry, using MAE and point cloud conditioning to
  mitigate the effects of occlusions and partial object information, ensuring
  accurate alignment with the source image's geometry and texture. To align each
  object with the scene, the alignment generation model computes the necessary
  transformations, allowing the generated meshes to be accurately placed and
  integrated into the scene's point cloud. Finally, CAST incorporates a
  physics-aware correction step that leverages a fine-grained relation graph to
  generate a constraint graph. This graph guides the optimization of object
  poses, ensuring physical consistency and spatial coherence. By utilizing Signed
  Distance Fields (SDF), the model effectively addresses issues such as
  occlusions, object penetration, and floating objects, ensuring that the
  generated scene accurately reflects real-world physical interactions. CAST can
  be leveraged in robotics, enabling efficient real-to-simulation workflows and
  providing realistic, scalable simulation environments for robotic systems.
paper_ai_summary: CAST is a method for reconstructing high-quality 3D scenes from single RGB images using GPT models, occlusion-aware 3D generation, and physics-aware correction with Signed Distance Fields.
paper_ai_keywords: CAPT GPT-based model occlusion-aware large-scale 3D generation model MAE point cloud conditioning alignment generation model constraint graph Signed Distance Fields (SDF)