title: VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document
    Understanding
summary: In recent years, notable advancements have been made in the domain of visual
  document understanding, with the prevailing architecture comprising a cascade
  of vision and language models. The text component can either be extracted
  explicitly with the use of external OCR models in OCR-based approaches, or
  alternatively, the vision model can be endowed with reading capabilities in
  OCR-free approaches. Typically, the queries to the model are input exclusively
  to the language component, necessitating the visual features to encompass the
  entire document. In this paper, we present VisFocus, an OCR-free method
  designed to better exploit the vision encoder's capacity by coupling it
  directly with the language prompt. To do so, we replace the down-sampling
  layers with layers that receive the input prompt and allow highlighting
  relevant parts of the document, while disregarding others. We pair the
  architecture enhancements with a novel pre-training task, using language
  masking on a snippet of the document text fed to the visual encoder in place of
  the prompt, to empower the model with focusing capabilities. Consequently,
  VisFocus learns to allocate its attention to text patches pertinent to the
  provided prompt. Our experiments demonstrate that this prompt-guided visual
  encoding approach significantly improves performance, achieving
  state-of-the-art results on various benchmarks.
paper_authors._id: 669e0b16de95466572dad018
paper_authors.name: Ofir Abramovich
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:51:16.580Z
paper_authors._id: 669e089bf433fc42beae9923
paper_authors.avatarUrl: /avatars/9e06da07e9b9ac0bdee0b28ccabeb0d1.svg
paper_authors.fullname: Ofir Abramovich
paper_authors.type: user
paper_authors.user: ofirab
paper_authors._id: 669e0b16de95466572dad019
paper_authors.name: Niv Nayman
paper_authors._id: 669e0b16de95466572dad01a
paper_authors.name: Sharon Fogel
paper_authors._id: 669e0b16de95466572dad01b
paper_authors.name: Inbal Lavi
paper_authors._id: 669e0b16de95466572dad01c
paper_authors.name: Ron Litman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:04:54.175Z
paper_authors._id: 65e70aa92db014a1b278a857
paper_authors.avatarUrl: /avatars/2f586a73408de49af10c1efe6abcb9cf.svg
paper_authors.fullname: Ron Litman
paper_authors.type: user
paper_authors.user: ronlitman
paper_authors._id: 669e0b16de95466572dad01d
paper_authors.name: Shahar Tsiper
paper_authors._id: 669e0b16de95466572dad01e
paper_authors.name: Royee Tichauer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:05:49.820Z
paper_authors._id: 64666d68dac127ac80f301c4
paper_authors.avatarUrl: /avatars/973848cc702e9bd9fed80b9b2678facf.svg
paper_authors.fullname: Royee Tichauer
paper_authors.type: user
paper_authors.user: royeet
paper_authors._id: 669e0b16de95466572dad01f
paper_authors.name: Srikar Appalaraju
paper_authors._id: 669e0b16de95466572dad020
paper_authors.name: Shai Mazor
paper_authors._id: 669e0b16de95466572dad021
paper_authors.name: R. Manmatha
paper_title: VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document
    Understanding
paper_summary: In recent years, notable advancements have been made in the domain of visual
  document understanding, with the prevailing architecture comprising a cascade
  of vision and language models. The text component can either be extracted
  explicitly with the use of external OCR models in OCR-based approaches, or
  alternatively, the vision model can be endowed with reading capabilities in
  OCR-free approaches. Typically, the queries to the model are input exclusively
  to the language component, necessitating the visual features to encompass the
  entire document. In this paper, we present VisFocus, an OCR-free method
  designed to better exploit the vision encoder's capacity by coupling it
  directly with the language prompt. To do so, we replace the down-sampling
  layers with layers that receive the input prompt and allow highlighting
  relevant parts of the document, while disregarding others. We pair the
  architecture enhancements with a novel pre-training task, using language
  masking on a snippet of the document text fed to the visual encoder in place of
  the prompt, to empower the model with focusing capabilities. Consequently,
  VisFocus learns to allocate its attention to text patches pertinent to the
  provided prompt. Our experiments demonstrate that this prompt-guided visual
  encoding approach significantly improves performance, achieving
  state-of-the-art results on various benchmarks.
paper_ai_summary: VisFocus enhances visual document understanding by integrating language prompts directly into the vision encoder, enabling focus on relevant text patches and improving performance across benchmarks.
paper_ai_keywords: vision encoder language prompt down-sampling layers language masking prompt-guided visual encoding