3D CoCa: Contrastive Learners are 3D Captioners
3D captioning, which aims to describe the content of 3D scenes in natural
language, remains highly challenging due to the inherent sparsity of point
clouds and weak cross-modal alignment in existing methods. To address these
challenges, we propose 3D CoCa, a novel unified framework that seamlessly
combines contrastive vision-language learning with 3D caption generation in a
single architecture. Our approach leverages a frozen CLIP vision-language
backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
capture geometric context, and a multi-modal decoder to generate descriptive
captions. Unlike prior two-stage methods that rely on explicit object
proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
shared feature space, eliminating the need for external detectors or
handcrafted proposals. This joint training paradigm yields stronger spatial
reasoning and richer semantic grounding by aligning 3D and textual
representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
https://github.com/AIGeeksGroup/3DCoCa.
67fe50f6b3a9473353b68941
Ting Huang
67fe50f6b3a9473353b68942
Zeyu Zhang
claimed_verified
2025-04-15T16:46:59.967Z
64ec877bb93654d4ca5c92e9
https://cdn-avatars.huggingface.co/v1/production/uploads/64ec877bb93654d4ca5c92e9/GvHk_KSdE9Rhnk_o-NaZX.jpeg
Zeyu Zhang
user
SteveZeyuZhang
67fe50f6b3a9473353b68943
Yemin Wang
67fe50f6b3a9473353b68944
Hao Tang
3D CoCa: Contrastive Learners are 3D Captioners
3D captioning, which aims to describe the content of 3D scenes in natural
language, remains highly challenging due to the inherent sparsity of point
clouds and weak cross-modal alignment in existing methods. To address these
challenges, we propose 3D CoCa, a novel unified framework that seamlessly
combines contrastive vision-language learning with 3D caption generation in a
single architecture. Our approach leverages a frozen CLIP vision-language
backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
capture geometric context, and a multi-modal decoder to generate descriptive
captions. Unlike prior two-stage methods that rely on explicit object
proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
shared feature space, eliminating the need for external detectors or
handcrafted proposals. This joint training paradigm yields stronger spatial
reasoning and richer semantic grounding by aligning 3D and textual
representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
https://github.com/AIGeeksGroup/3DCoCa.
3D CoCa is a unified framework combining contrastive vision-language learning with 3D caption generation, using a CLIP backbone, spatially-aware 3D scene encoder, and multi-modal decoder for improved spatial reasoning and semantic grounding.
contrastive vision-language learning 3D caption generation CLIP spatially-aware 3D scene encoder multi-modal decoder ScanRefer Nr3D CIDEr IoU