CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
Recovering high-quality 3D scenes from a single RGB image is a challenging
task in computer graphics. Current methods often struggle with domain-specific
limitations or low-quality object generation. To address these, we propose CAST
(Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel
method for 3D scene reconstruction and recovery. CAST starts by extracting
object-level 2D segmentation and relative depth information from the input
image, followed by using a GPT-based model to analyze inter-object spatial
relationships. This enables the understanding of how objects relate to each
other within the scene, ensuring more coherent reconstruction. CAST then
employs an occlusion-aware large-scale 3D generation model to independently
generate each object's full geometry, using MAE and point cloud conditioning to
mitigate the effects of occlusions and partial object information, ensuring
accurate alignment with the source image's geometry and texture. To align each
object with the scene, the alignment generation model computes the necessary
transformations, allowing the generated meshes to be accurately placed and
integrated into the scene's point cloud. Finally, CAST incorporates a
physics-aware correction step that leverages a fine-grained relation graph to
generate a constraint graph. This graph guides the optimization of object
poses, ensuring physical consistency and spatial coherence. By utilizing Signed
Distance Fields (SDF), the model effectively addresses issues such as
occlusions, object penetration, and floating objects, ensuring that the
generated scene accurately reflects real-world physical interactions. CAST can
be leveraged in robotics, enabling efficient real-to-simulation workflows and
providing realistic, scalable simulation environments for robotic systems.
67b60a5c92649eb787234701
Kaixin Yao
67b60a5c92649eb787234702
Longwen Zhang
claimed_verified
2025-07-31T08:16:52.310Z
636d12455aaed143cd665607
https://cdn-avatars.huggingface.co/v1/production/uploads/1679399015950-636d12455aaed143cd665607.png
ZLW
user
ZarkLngeW
67b60a5c92649eb787234703
Xinhao Yan
67b60a5c92649eb787234704
Yan Zeng
67b60a5c92649eb787234705
Qixuan Zhang
67b60a5c92649eb787234706
Lan Xu
67b60a5c92649eb787234707
Wei Yang
67b60a5c92649eb787234708
Jiayuan Gu
67b60a5c92649eb787234709
Jingyi Yu
CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
Recovering high-quality 3D scenes from a single RGB image is a challenging
task in computer graphics. Current methods often struggle with domain-specific
limitations or low-quality object generation. To address these, we propose CAST
(Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel
method for 3D scene reconstruction and recovery. CAST starts by extracting
object-level 2D segmentation and relative depth information from the input
image, followed by using a GPT-based model to analyze inter-object spatial
relationships. This enables the understanding of how objects relate to each
other within the scene, ensuring more coherent reconstruction. CAST then
employs an occlusion-aware large-scale 3D generation model to independently
generate each object's full geometry, using MAE and point cloud conditioning to
mitigate the effects of occlusions and partial object information, ensuring
accurate alignment with the source image's geometry and texture. To align each
object with the scene, the alignment generation model computes the necessary
transformations, allowing the generated meshes to be accurately placed and
integrated into the scene's point cloud. Finally, CAST incorporates a
physics-aware correction step that leverages a fine-grained relation graph to
generate a constraint graph. This graph guides the optimization of object
poses, ensuring physical consistency and spatial coherence. By utilizing Signed
Distance Fields (SDF), the model effectively addresses issues such as
occlusions, object penetration, and floating objects, ensuring that the
generated scene accurately reflects real-world physical interactions. CAST can
be leveraged in robotics, enabling efficient real-to-simulation workflows and
providing realistic, scalable simulation environments for robotic systems.
CAST is a method for reconstructing high-quality 3D scenes from single RGB images using GPT models, occlusion-aware 3D generation, and physics-aware correction with Signed Distance Fields.
CAPT GPT-based model occlusion-aware large-scale 3D generation model MAE point cloud conditioning alignment generation model constraint graph Signed Distance Fields (SDF)