Contrastive Localized Language-Image Pre-Training
Contrastive Language-Image Pre-training (CLIP) has been a celebrated method
for training vision encoders to generate image/text representations
facilitating various applications. Recently, CLIP has been widely adopted as
the vision backbone of multimodal large language models (MLLMs) to connect
image inputs for language interactions. The success of CLIP as a
vision-language foundation model relies on aligning web-crawled noisy text
annotations at image levels. Nevertheless, such criteria may become
insufficient for downstream tasks in need of fine-grained vision
representations, especially when region-level understanding is demanding for
MLLMs. In this paper, we improve the localization capability of CLIP with
several advances. We propose a pre-training method called Contrastive Localized
Language-Image Pre-training (CLOC) by complementing CLIP with region-text
contrastive loss and modules. We formulate a new concept, promptable
embeddings, of which the encoder produces image embeddings easy to transform
into region representations given spatial hints. To support large-scale
pre-training, we design a visually-enriched and spatially-localized captioning
framework to effectively generate region-text pseudo-labels at scale. By
scaling up to billions of annotated images, CLOC enables high-quality regional
embeddings for image region recognition and retrieval tasks, and can be a
drop-in replacement of CLIP to enhance MLLMs, especially on referring and
grounding tasks.
66ff590b39485e454983f18e
Hong-You Chen
claimed_verified
2024-10-04T07:30:57.578Z
66fb920d63d612c629bfea36
/avatars/86f5ccd4a15e8394792b422fefa64b69.svg
Chen
user
Hong-You
66ff590b39485e454983f18f
Zhengfeng Lai
claimed_verified
2024-10-04T07:30:55.894Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
66ff590b39485e454983f190
Haotian Zhang
claimed_verified
2024-10-04T07:30:59.170Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
66ff590b39485e454983f191
Xinze Wang
66ff590b39485e454983f192
Marcin Eichner
66ff590b39485e454983f193
Keen You
66ff590b39485e454983f194
Meng Cao
66ff590b39485e454983f195
Bowen Zhang
66ff590b39485e454983f196
Yinfei Yang
66ff590b39485e454983f197
Zhe Gan
Contrastive Localized Language-Image Pre-Training
Contrastive Language-Image Pre-training (CLIP) has been a celebrated method
for training vision encoders to generate image/text representations
facilitating various applications. Recently, CLIP has been widely adopted as
the vision backbone of multimodal large language models (MLLMs) to connect
image inputs for language interactions. The success of CLIP as a
vision-language foundation model relies on aligning web-crawled noisy text
annotations at image levels. Nevertheless, such criteria may become
insufficient for downstream tasks in need of fine-grained vision
representations, especially when region-level understanding is demanding for
MLLMs. In this paper, we improve the localization capability of CLIP with
several advances. We propose a pre-training method called Contrastive Localized
Language-Image Pre-training (CLOC) by complementing CLIP with region-text
contrastive loss and modules. We formulate a new concept, promptable
embeddings, of which the encoder produces image embeddings easy to transform
into region representations given spatial hints. To support large-scale
pre-training, we design a visually-enriched and spatially-localized captioning
framework to effectively generate region-text pseudo-labels at scale. By
scaling up to billions of annotated images, CLOC enables high-quality regional
embeddings for image region recognition and retrieval tasks, and can be a
drop-in replacement of CLIP to enhance MLLMs, especially on referring and
grounding tasks.
CLOC enhances CLIP's localization capabilities by introducing region-text contrastive loss and promptable embeddings, improving regional image representation for multimodal large language models.
Contrastive Language-Image Pre-training (CLIP) multimodal large language models (MLLMs) region-text contrastive loss promptable embeddings region representations visually-enriched captioning framework regional embeddings image region recognition retrieval tasks