title: VCR: Visual Caption Restoration
summary: We introduce Visual Caption Restoration (VCR), a novel vision-language task
  that challenges models to accurately restore partially obscured texts using
  pixel-level hints within images. This task stems from the observation that text
  embedded in images is intrinsically different from common visual elements and
  natural language due to the need to align the modalities of vision, text, and
  text embedded in images. While numerous works have integrated text embedded in
  images into visual question-answering tasks, approaches to these tasks
  generally rely on optical character recognition or masked language modeling,
  thus reducing the task to mainly text-based processing. However, text-based
  processing becomes ineffective in VCR as accurate text restoration depends on
  the combined information from provided images, context, and subtle cues from
  the tiny exposed areas of masked texts. We develop a pipeline to generate
  synthetic images for the VCR task using image-caption pairs, with adjustable
  caption visibility to control the task difficulty. With this pipeline, we
  construct a dataset for VCR called VCR-Wiki using images with captions from
  Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
  hard split variants. Our results reveal that current vision language models
  significantly lag behind human performance in the VCR task, and merely
  fine-tuning the models on our dataset does not lead to notable improvements. We
  release VCR-Wiki and the data construction code to facilitate future research.
paper_authors._id: 6667b702dd968ec963cb9dee
paper_authors.name: Tianyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T18:36:20.440Z
paper_authors._id: 6452d79149b6b9a2383b5775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T28lP0kE7PZIGzJjhSpSx.jpeg
paper_authors.fullname: Tianyu Zhang
paper_authors.type: user
paper_authors.user: TianyuZhang
paper_authors._id: 6667b702dd968ec963cb9def
paper_authors.name: Suyuchen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T06:59:10.569Z
paper_authors._id: 62bb1e0f3ff437e49a3088e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/bcUQmH8tKfI6DIWH9IcYp.jpeg
paper_authors.fullname: Suyuchen Wang
paper_authors.type: user
paper_authors.user: sheryc
paper_authors._id: 6667b702dd968ec963cb9df0
paper_authors.name: Lu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T21:29:05.165Z
paper_authors._id: 658fbec068d0b76331a81e38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658fbec068d0b76331a81e38/E-1fyTCM4eltkWOXwWU5k.jpeg
paper_authors.fullname: Lu Li
paper_authors.type: user
paper_authors.user: luli2949
paper_authors._id: 6667b702dd968ec963cb9df1
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:15.807Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6667b702dd968ec963cb9df2
paper_authors.name: Perouz Taslakian
paper_authors._id: 6667b702dd968ec963cb9df3
paper_authors.name: Sai Rajeswar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:47:30.661Z
paper_authors._id: 642f99079b2484d7d857341b
paper_authors.avatarUrl: /avatars/01965cc5a5dbe9c08025a51973462a6a.svg
paper_authors.fullname: Sai Rajeswar
paper_authors.type: user
paper_authors.user: rajeswarsai
paper_authors._id: 6667b702dd968ec963cb9df4
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:33:20.052Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 6667b702dd968ec963cb9df5
paper_authors.name: Bang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:24:35.896Z
paper_authors._id: 654a97282d2fcd6bf2851173
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
paper_authors.fullname: Bang Liu
paper_authors.type: user
paper_authors.user: Bang-UdeM-Mila
paper_authors._id: 6667b702dd968ec963cb9df6
paper_authors.name: Yoshua Bengio
paper_title: VCR: Visual Caption Restoration
paper_summary: We introduce Visual Caption Restoration (VCR), a novel vision-language task
  that challenges models to accurately restore partially obscured texts using
  pixel-level hints within images. This task stems from the observation that text
  embedded in images is intrinsically different from common visual elements and
  natural language due to the need to align the modalities of vision, text, and
  text embedded in images. While numerous works have integrated text embedded in
  images into visual question-answering tasks, approaches to these tasks
  generally rely on optical character recognition or masked language modeling,
  thus reducing the task to mainly text-based processing. However, text-based
  processing becomes ineffective in VCR as accurate text restoration depends on
  the combined information from provided images, context, and subtle cues from
  the tiny exposed areas of masked texts. We develop a pipeline to generate
  synthetic images for the VCR task using image-caption pairs, with adjustable
  caption visibility to control the task difficulty. With this pipeline, we
  construct a dataset for VCR called VCR-Wiki using images with captions from
  Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
  hard split variants. Our results reveal that current vision language models
  significantly lag behind human performance in the VCR task, and merely
  fine-tuning the models on our dataset does not lead to notable improvements. We
  release VCR-Wiki and the data construction code to facilitate future research.
paper_ai_keywords: Visual Caption Restoration (VCR) vision-language task partially obscured texts pixel-level hints embedded in images align the modalities optical character recognition masked language modeling synthetic images image-caption pairs caption visibility VCR-Wiki task difficulty vision language models