title: Parrot Captions Teach CLIP to Spot Text
summary: Despite CLIP being the foundation model in numerous vision-language
  applications, the CLIP suffers from a severe text spotting bias. Such bias
  causes CLIP models to `Parrot' the visual text embedded within images while
  disregarding the authentic visual semantics. We uncover that in the most
  popular image-text dataset LAION-2B, the captions also densely parrot (spell)
  the text embedded in images. Our analysis shows that around 50\% of
  images are embedded with visual text content, and 90\% of their
  captions more or less parrot the visual text. Based on such observation, we
  thoroughly inspect the different release d versions of CLIP models and verify
  that the visual text is the dominant factor in measuring the LAION-style
  image-text similarity for these models. To examine whether these parrot
  captions shape the text spotting bias, we train a series of CLIP models with
  LAION subsets curated by different parrot-caption-oriented criteria. We show
  that training with parrot captions easily shapes such bias but harms the
  expected visual-language representation learning in CLIP models. This suggests
  that it is urgent to revisit either the design of CLIP-like models or the
  existing image-text dataset curation pipeline built on CLIP score filtering.
paper_authors._id: 6588dedd6cf9325ae6cf3587
paper_authors.name: Yiqi Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:49:08.913Z
paper_authors._id: 630df8a8c6b1d1bccb7ce7ce
paper_authors.avatarUrl: /avatars/a17aa724a9ba2c186412d66a21a60597.svg
paper_authors.fullname: linyiqi
paper_authors.type: user
paper_authors.user: linyq
paper_authors._id: 6588dedd6cf9325ae6cf3588
paper_authors.name: Conghui He
paper_authors._id: 6588dedd6cf9325ae6cf3589
paper_authors.name: Alex Jinpeng Wang
paper_authors._id: 6588dedd6cf9325ae6cf358a
paper_authors.name: Bin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:06:26.082Z
paper_authors._id: 63ae9ff5557befe297a76f90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: wanderkid
paper_authors._id: 6588dedd6cf9325ae6cf358b
paper_authors.name: Weijia Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:50:36.541Z
paper_authors._id: 66d5b56c77a026c3d2086a79
paper_authors.avatarUrl: /avatars/45da07fd82fd455955faa05b27a6393f.svg
paper_authors.fullname: Weijia Li
paper_authors.type: user
paper_authors.user: liweijia
paper_authors._id: 6588dedd6cf9325ae6cf358c
paper_authors.name: Mike Zheng Shou
paper_title: Parrot Captions Teach CLIP to Spot Text
paper_summary: Despite CLIP being the foundation model in numerous vision-language
  applications, the CLIP suffers from a severe text spotting bias. Such bias
  causes CLIP models to `Parrot' the visual text embedded within images while
  disregarding the authentic visual semantics. We uncover that in the most
  popular image-text dataset LAION-2B, the captions also densely parrot (spell)
  the text embedded in images. Our analysis shows that around 50\% of
  images are embedded with visual text content, and 90\% of their
  captions more or less parrot the visual text. Based on such observation, we
  thoroughly inspect the different release d versions of CLIP models and verify
  that the visual text is the dominant factor in measuring the LAION-style
  image-text similarity for these models. To examine whether these parrot
  captions shape the text spotting bias, we train a series of CLIP models with
  LAION subsets curated by different parrot-caption-oriented criteria. We show
  that training with parrot captions easily shapes such bias but harms the
  expected visual-language representation learning in CLIP models. This suggests
  that it is urgent to revisit either the design of CLIP-like models or the
  existing image-text dataset curation pipeline built on CLIP score filtering.
paper_ai_keywords: CLIP visual-language applications text spotting bias visual semantics LAION-2B text embeddings multimodal image-text dataset visual text content captioning CLIP models image-text reduction embedding inversion image-text similarity visual-language representation learning parrot captions