title: 3D CoCa: Contrastive Learners are 3D Captioners
summary: 3D captioning, which aims to describe the content of 3D scenes in natural
  language, remains highly challenging due to the inherent sparsity of point
  clouds and weak cross-modal alignment in existing methods. To address these
  challenges, we propose 3D CoCa, a novel unified framework that seamlessly
  combines contrastive vision-language learning with 3D caption generation in a
  single architecture. Our approach leverages a frozen CLIP vision-language
  backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
  capture geometric context, and a multi-modal decoder to generate descriptive
  captions. Unlike prior two-stage methods that rely on explicit object
  proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
  shared feature space, eliminating the need for external detectors or
  handcrafted proposals. This joint training paradigm yields stronger spatial
  reasoning and richer semantic grounding by aligning 3D and textual
  representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
  demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
  10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
  https://github.com/AIGeeksGroup/3DCoCa.
paper_authors._id: 67fe50f6b3a9473353b68941
paper_authors.name: Ting Huang
paper_authors._id: 67fe50f6b3a9473353b68942
paper_authors.name: Zeyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T16:46:59.967Z
paper_authors._id: 64ec877bb93654d4ca5c92e9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ec877bb93654d4ca5c92e9/GvHk_KSdE9Rhnk_o-NaZX.jpeg
paper_authors.fullname: Zeyu Zhang
paper_authors.type: user
paper_authors.user: SteveZeyuZhang
paper_authors._id: 67fe50f6b3a9473353b68943
paper_authors.name: Yemin Wang
paper_authors._id: 67fe50f6b3a9473353b68944
paper_authors.name: Hao Tang
paper_title: 3D CoCa: Contrastive Learners are 3D Captioners
paper_summary: 3D captioning, which aims to describe the content of 3D scenes in natural
  language, remains highly challenging due to the inherent sparsity of point
  clouds and weak cross-modal alignment in existing methods. To address these
  challenges, we propose 3D CoCa, a novel unified framework that seamlessly
  combines contrastive vision-language learning with 3D caption generation in a
  single architecture. Our approach leverages a frozen CLIP vision-language
  backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
  capture geometric context, and a multi-modal decoder to generate descriptive
  captions. Unlike prior two-stage methods that rely on explicit object
  proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
  shared feature space, eliminating the need for external detectors or
  handcrafted proposals. This joint training paradigm yields stronger spatial
  reasoning and richer semantic grounding by aligning 3D and textual
  representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
  demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
  10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
  https://github.com/AIGeeksGroup/3DCoCa.
paper_ai_summary: 3D CoCa is a unified framework combining contrastive vision-language learning with 3D caption generation, using a CLIP backbone, spatially-aware 3D scene encoder, and multi-modal decoder for improved spatial reasoning and semantic grounding.
paper_ai_keywords: contrastive vision-language learning 3D caption generation CLIP spatially-aware 3D scene encoder multi-modal decoder ScanRefer Nr3D CIDEr IoU