No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard Negatives via CLIP Knowledge and LLMs
<|reference_start|>No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard Negatives via CLIP Knowledge and LLMs: In this study, we explore an alternative approach to enhance contrastive text-image-3D alignment in the absence of textual descriptions for 3D objects. We introduce two unsupervised methods, $I2I$ and $(I2L)^2$, which leverage CLIP knowledge about textual and 2D data to compute the neural perceived similarity between two 3D samples. We employ the proposed methods to mine 3D hard negatives, establishing a multimodal contrastive pipeline with hard negative weighting via a custom loss function. We train on different configurations of the proposed hard negative mining approach, and we evaluate the accuracy of our models in 3D classification and on the cross-modal retrieval benchmark, testing image-to-shape and shape-to-image retrieval. Results demonstrate that our approach, even without explicit text alignment, achieves comparable or superior performance on zero-shot and standard 3D classification, while significantly improving both image-to-shape and shape-to-image retrieval compared to previous methods.<|reference_end|>
@article{sbrolli2024no,
    title={No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard
  Negatives via CLIP Knowledge and LLMs},
    author={Cristian Sbrolli and Matteo Matteucci},
    journal={arXiv preprint arXiv:2406.02202},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02202},
    primaryClass={cs.CV cs.AI}
}