title: Contrastive Localized Language-Image Pre-Training
summary: Contrastive Language-Image Pre-training (CLIP) has been a celebrated method
  for training vision encoders to generate image/text representations
  facilitating various applications. Recently, CLIP has been widely adopted as
  the vision backbone of multimodal large language models (MLLMs) to connect
  image inputs for language interactions. The success of CLIP as a
  vision-language foundation model relies on aligning web-crawled noisy text
  annotations at image levels. Nevertheless, such criteria may become
  insufficient for downstream tasks in need of fine-grained vision
  representations, especially when region-level understanding is demanding for
  MLLMs. In this paper, we improve the localization capability of CLIP with
  several advances. We propose a pre-training method called Contrastive Localized
  Language-Image Pre-training (CLOC) by complementing CLIP with region-text
  contrastive loss and modules. We formulate a new concept, promptable
  embeddings, of which the encoder produces image embeddings easy to transform
  into region representations given spatial hints. To support large-scale
  pre-training, we design a visually-enriched and spatially-localized captioning
  framework to effectively generate region-text pseudo-labels at scale. By
  scaling up to billions of annotated images, CLOC enables high-quality regional
  embeddings for image region recognition and retrieval tasks, and can be a
  drop-in replacement of CLIP to enhance MLLMs, especially on referring and
  grounding tasks.
paper_authors._id: 66ff590b39485e454983f18e
paper_authors.name: Hong-You Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:30:57.578Z
paper_authors._id: 66fb920d63d612c629bfea36
paper_authors.avatarUrl: /avatars/86f5ccd4a15e8394792b422fefa64b69.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Hong-You
paper_authors._id: 66ff590b39485e454983f18f
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:30:55.894Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 66ff590b39485e454983f190
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:30:59.170Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 66ff590b39485e454983f191
paper_authors.name: Xinze Wang
paper_authors._id: 66ff590b39485e454983f192
paper_authors.name: Marcin Eichner
paper_authors._id: 66ff590b39485e454983f193
paper_authors.name: Keen You
paper_authors._id: 66ff590b39485e454983f194
paper_authors.name: Meng Cao
paper_authors._id: 66ff590b39485e454983f195
paper_authors.name: Bowen Zhang
paper_authors._id: 66ff590b39485e454983f196
paper_authors.name: Yinfei Yang
paper_authors._id: 66ff590b39485e454983f197
paper_authors.name: Zhe Gan
paper_title: Contrastive Localized Language-Image Pre-Training
paper_summary: Contrastive Language-Image Pre-training (CLIP) has been a celebrated method
  for training vision encoders to generate image/text representations
  facilitating various applications. Recently, CLIP has been widely adopted as
  the vision backbone of multimodal large language models (MLLMs) to connect
  image inputs for language interactions. The success of CLIP as a
  vision-language foundation model relies on aligning web-crawled noisy text
  annotations at image levels. Nevertheless, such criteria may become
  insufficient for downstream tasks in need of fine-grained vision
  representations, especially when region-level understanding is demanding for
  MLLMs. In this paper, we improve the localization capability of CLIP with
  several advances. We propose a pre-training method called Contrastive Localized
  Language-Image Pre-training (CLOC) by complementing CLIP with region-text
  contrastive loss and modules. We formulate a new concept, promptable
  embeddings, of which the encoder produces image embeddings easy to transform
  into region representations given spatial hints. To support large-scale
  pre-training, we design a visually-enriched and spatially-localized captioning
  framework to effectively generate region-text pseudo-labels at scale. By
  scaling up to billions of annotated images, CLOC enables high-quality regional
  embeddings for image region recognition and retrieval tasks, and can be a
  drop-in replacement of CLIP to enhance MLLMs, especially on referring and
  grounding tasks.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) multimodal large language models (MLLMs) vision backbone vision encoders image/text representations linguistic-visual representations fast MLML representation region-level understanding localization capability Contrastive Localized Language-Image Pre-training (CLOC) region-text promptable embeddings image embeddings region representations visually-enriched spatially-localized captioning region-text pseudo-labels regional embeddings image region recognition image region retrieval tasks drop-in replacement