title: FG-CLIP: Fine-Grained Visual and Textual Alignment
summary: Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks
  such as image-text retrieval and zero-shot classification but struggles with
  fine-grained understanding due to its focus on coarse-grained short captions.
  To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances
  fine-grained understanding through three key innovations. First, we leverage
  large multimodal models to generate 1.6 billion long caption-image pairs for
  capturing global-level semantic details. Second, a high-quality dataset is
  constructed with 12 million images and 40 million region-specific bounding
  boxes aligned with detailed captions to ensure precise, context-rich
  representations. Third, 10 million hard fine-grained negative samples are
  incorporated to improve the model's ability to distinguish subtle semantic
  differences. Corresponding training methods are meticulously designed for these
  data. Extensive experiments demonstrate that FG-CLIP outperforms the original
  CLIP and other state-of-the-art methods across various downstream tasks,
  including fine-grained understanding, open-vocabulary object detection,
  image-text retrieval, and general multimodal benchmarks. These results
  highlight FG-CLIP's effectiveness in capturing fine-grained image details and
  improving overall model performance. The related data, code, and models are
  available at https://github.com/360CVGroup/FG-CLIP.
paper_authors._id: 681da6375f701833274a0d21
paper_authors.name: Chunyu Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:21:03.296Z
paper_authors._id: 6621e591c50869c1e91a1639
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6621e591c50869c1e91a1639/L_PoEn2BRAJcnWZX-JebR.jpeg
paper_authors.fullname: Chunyu Xie
paper_authors.type: user
paper_authors.user: xiechunyu
paper_authors._id: 681da6375f701833274a0d22
paper_authors.name: Bin Wang
paper_authors._id: 681da6375f701833274a0d23
paper_authors.name: Fanjing Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:58:42.881Z
paper_authors._id: 632c098b456c31252774e7c5
paper_authors.avatarUrl: /avatars/e3720d2fcb69d93c8d5aa5f50aab5f0e.svg
paper_authors.fullname: kong
paper_authors.type: user
paper_authors.user: fanjing
paper_authors._id: 681da6375f701833274a0d24
paper_authors.name: Jincheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:58:50.538Z
paper_authors._id: 65b793b374698ba5a815bf4f
paper_authors.avatarUrl: /avatars/44a7e694a5089dbc773018111270ac26.svg
paper_authors.fullname: Jincheng Li
paper_authors.type: user
paper_authors.user: jinchenglijc
paper_authors._id: 681da6375f701833274a0d25
paper_authors.name: Dawei Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:58:56.445Z
paper_authors._id: 659b8576999b82db2ad8a398
paper_authors.avatarUrl: /avatars/2ec7663e25e4a0238819818e69d9a5bd.svg
paper_authors.fullname: Liang
paper_authors.type: user
paper_authors.user: DaweiLiang
paper_authors._id: 681da6375f701833274a0d26
paper_authors.name: Gengshen Zhang
paper_authors._id: 681da6375f701833274a0d27
paper_authors.name: Dawei Leng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-09T07:00:10.249Z
paper_authors._id: 649935abbe8fd92c27ab1ed8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
paper_authors.fullname: David Leon
paper_authors.type: user
paper_authors.user: DavidLeon
paper_authors._id: 681da6375f701833274a0d28
paper_authors.name: Yuhui Yin
paper_title: FG-CLIP: Fine-Grained Visual and Textual Alignment
paper_summary: Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks
  such as image-text retrieval and zero-shot classification but struggles with
  fine-grained understanding due to its focus on coarse-grained short captions.
  To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances
  fine-grained understanding through three key innovations. First, we leverage
  large multimodal models to generate 1.6 billion long caption-image pairs for
  capturing global-level semantic details. Second, a high-quality dataset is
  constructed with 12 million images and 40 million region-specific bounding
  boxes aligned with detailed captions to ensure precise, context-rich
  representations. Third, 10 million hard fine-grained negative samples are
  incorporated to improve the model's ability to distinguish subtle semantic
  differences. Corresponding training methods are meticulously designed for these
  data. Extensive experiments demonstrate that FG-CLIP outperforms the original
  CLIP and other state-of-the-art methods across various downstream tasks,
  including fine-grained understanding, open-vocabulary object detection,
  image-text retrieval, and general multimodal benchmarks. These results
  highlight FG-CLIP's effectiveness in capturing fine-grained image details and
  improving overall model performance. The related data, code, and models are
  available at https://github.com/360CVGroup/FG-CLIP.
paper_ai_summary: FG-CLIP enhances fine-grained understanding in multimodal tasks by leveraging large multimodal models, a high-quality dataset with detailed captions, and hard fine-grained negative samples.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) Fine-Grained CLIP (FG-CLIP) multimodal models caption-image pairs global-level semantic details high-quality dataset region-specific bounding boxes detailed captions fine-grained negative samples fine-grained understanding open-vocabulary object detection image-text retrieval multimodal benchmarks