title: CoLLM: A Large Language Model for Composed Image Retrieval
summary: Composed Image Retrieval (CIR) is a complex task that aims to retrieve images
  based on a multimodal query. Typical training data consists of triplets
  containing a reference image, a textual description of desired modifications,
  and the target image, which are expensive and time-consuming to acquire. The
  scarcity of CIR datasets has led to zero-shot approaches utilizing synthetic
  triplets or leveraging vision-language models (VLMs) with ubiquitous
  web-crawled image-caption pairs. However, these methods have significant
  limitations: synthetic triplets suffer from limited scale, lack of diversity,
  and unnatural modification text, while image-caption pairs hinder joint
  embedding learning of the multimodal query due to the absence of triplet data.
  Moreover, existing approaches struggle with complex and nuanced modification
  texts that demand sophisticated fusion and understanding of vision and language
  modalities. We present CoLLM, a one-stop framework that effectively addresses
  these limitations. Our approach generates triplets on-the-fly from
  image-caption pairs, enabling supervised training without manual annotation. We
  leverage Large Language Models (LLMs) to generate joint embeddings of reference
  images and modification texts, facilitating deeper multimodal fusion.
  Additionally, we introduce Multi-Text CIR (MTCIR), a large-scale dataset
  comprising 3.4M samples, and refine existing CIR benchmarks (CIRR and
  Fashion-IQ) to enhance evaluation reliability. Experimental results demonstrate
  that CoLLM achieves state-of-the-art performance across multiple CIR benchmarks
  and settings. MTCIR yields competitive results, with up to 15% performance
  improvement. Our refined benchmarks provide more reliable evaluation metrics
  for CIR models, contributing to the advancement of this important field.
paper_authors._id: 67e35e4cff080b9ee71e3295
paper_authors.name: Chuong Huynh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:39.247Z
paper_authors._id: 63a4d196cde2b28f82a56bd9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4d196cde2b28f82a56bd9/iqVFOtDteRMUScFGRcx0L.png
paper_authors.fullname: Chuong Huynh
paper_authors.type: user
paper_authors.user: chuonghm
paper_authors._id: 67e35e4cff080b9ee71e3296
paper_authors.name: Jinyu Yang
paper_authors._id: 67e35e4cff080b9ee71e3297
paper_authors.name: Ashish Tawari
paper_authors._id: 67e35e4cff080b9ee71e3298
paper_authors.name: Mubarak Shah
paper_authors._id: 67e35e4cff080b9ee71e3299
paper_authors.name: Son Tran
paper_authors._id: 67e35e4cff080b9ee71e329a
paper_authors.name: Raffay Hamid
paper_authors._id: 67e35e4cff080b9ee71e329b
paper_authors.name: Trishul Chilimbi
paper_authors._id: 67e35e4cff080b9ee71e329c
paper_authors.name: Abhinav Shrivastava
paper_title: CoLLM: A Large Language Model for Composed Image Retrieval
paper_summary: Composed Image Retrieval (CIR) is a complex task that aims to retrieve images
  based on a multimodal query. Typical training data consists of triplets
  containing a reference image, a textual description of desired modifications,
  and the target image, which are expensive and time-consuming to acquire. The
  scarcity of CIR datasets has led to zero-shot approaches utilizing synthetic
  triplets or leveraging vision-language models (VLMs) with ubiquitous
  web-crawled image-caption pairs. However, these methods have significant
  limitations: synthetic triplets suffer from limited scale, lack of diversity,
  and unnatural modification text, while image-caption pairs hinder joint
  embedding learning of the multimodal query due to the absence of triplet data.
  Moreover, existing approaches struggle with complex and nuanced modification
  texts that demand sophisticated fusion and understanding of vision and language
  modalities. We present CoLLM, a one-stop framework that effectively addresses
  these limitations. Our approach generates triplets on-the-fly from
  image-caption pairs, enabling supervised training without manual annotation. We
  leverage Large Language Models (LLMs) to generate joint embeddings of reference
  images and modification texts, facilitating deeper multimodal fusion.
  Additionally, we introduce Multi-Text CIR (MTCIR), a large-scale dataset
  comprising 3.4M samples, and refine existing CIR benchmarks (CIRR and
  Fashion-IQ) to enhance evaluation reliability. Experimental results demonstrate
  that CoLLM achieves state-of-the-art performance across multiple CIR benchmarks
  and settings. MTCIR yields competitive results, with up to 15% performance
  improvement. Our refined benchmarks provide more reliable evaluation metrics
  for CIR models, contributing to the advancement of this important field.
paper_ai_summary: CoLLM addresses challenges in Composed Image Retrieval by generating triplets using LLMs and introducing a large dataset, achieving top performance across benchmarks.
paper_ai_keywords: Large Language Models LLNs Multi-Text CIR MTCIR joint embeddings multimodal fusion CIR benchmarks CIRR Fashion-IQ