Parrot Captions Teach CLIP to Spot Text
Despite CLIP being the foundation model in numerous vision-language
applications, the CLIP suffers from a severe text spotting bias. Such bias
causes CLIP models to `Parrot' the visual text embedded within images while
disregarding the authentic visual semantics. We uncover that in the most
popular image-text dataset LAION-2B, the captions also densely parrot (spell)
the text embedded in images. Our analysis shows that around 50\% of
images are embedded with visual text content, and 90\% of their
captions more or less parrot the visual text. Based on such observation, we
thoroughly inspect the different release d versions of CLIP models and verify
that the visual text is the dominant factor in measuring the LAION-style
image-text similarity for these models. To examine whether these parrot
captions shape the text spotting bias, we train a series of CLIP models with
LAION subsets curated by different parrot-caption-oriented criteria. We show
that training with parrot captions easily shapes such bias but harms the
expected visual-language representation learning in CLIP models. This suggests
that it is urgent to revisit either the design of CLIP-like models or the
existing image-text dataset curation pipeline built on CLIP score filtering.
6588dedd6cf9325ae6cf3587
Yiqi Lin
claimed_verified
2024-06-27T09:49:08.913Z
630df8a8c6b1d1bccb7ce7ce
/avatars/a17aa724a9ba2c186412d66a21a60597.svg
linyiqi
user
linyq
6588dedd6cf9325ae6cf3588
Conghui He
6588dedd6cf9325ae6cf3589
Alex Jinpeng Wang
6588dedd6cf9325ae6cf358a
Bin Wang
claimed_verified
2024-04-02T11:06:26.082Z
63ae9ff5557befe297a76f90
https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
Bin Wang
user
wanderkid
6588dedd6cf9325ae6cf358b
Weijia Li
claimed_verified
2024-09-03T07:50:36.541Z
66d5b56c77a026c3d2086a79
/avatars/45da07fd82fd455955faa05b27a6393f.svg
Weijia Li
user
liweijia
6588dedd6cf9325ae6cf358c
Mike Zheng Shou
Parrot Captions Teach CLIP to Spot Text
Despite CLIP being the foundation model in numerous vision-language
applications, the CLIP suffers from a severe text spotting bias. Such bias
causes CLIP models to `Parrot' the visual text embedded within images while
disregarding the authentic visual semantics. We uncover that in the most
popular image-text dataset LAION-2B, the captions also densely parrot (spell)
the text embedded in images. Our analysis shows that around 50\% of
images are embedded with visual text content, and 90\% of their
captions more or less parrot the visual text. Based on such observation, we
thoroughly inspect the different release d versions of CLIP models and verify
that the visual text is the dominant factor in measuring the LAION-style
image-text similarity for these models. To examine whether these parrot
captions shape the text spotting bias, we train a series of CLIP models with
LAION subsets curated by different parrot-caption-oriented criteria. We show
that training with parrot captions easily shapes such bias but harms the
expected visual-language representation learning in CLIP models. This suggests
that it is urgent to revisit either the design of CLIP-like models or the
existing image-text dataset curation pipeline built on CLIP score filtering.
CLIP models exhibit a text spotting bias in vision-language applications due to excessive reliance on visual text within images in datasets like LAION-2B.
CLIP vision-language applications text spotting bias LAION-2B visual text image-text similarity visual-language representation learning CLIP-like models image-text dataset curation