VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document
  Understanding
In recent years, notable advancements have been made in the domain of visual
document understanding, with the prevailing architecture comprising a cascade
of vision and language models. The text component can either be extracted
explicitly with the use of external OCR models in OCR-based approaches, or
alternatively, the vision model can be endowed with reading capabilities in
OCR-free approaches. Typically, the queries to the model are input exclusively
to the language component, necessitating the visual features to encompass the
entire document. In this paper, we present VisFocus, an OCR-free method
designed to better exploit the vision encoder's capacity by coupling it
directly with the language prompt. To do so, we replace the down-sampling
layers with layers that receive the input prompt and allow highlighting
relevant parts of the document, while disregarding others. We pair the
architecture enhancements with a novel pre-training task, using language
masking on a snippet of the document text fed to the visual encoder in place of
the prompt, to empower the model with focusing capabilities. Consequently,
VisFocus learns to allocate its attention to text patches pertinent to the
provided prompt. Our experiments demonstrate that this prompt-guided visual
encoding approach significantly improves performance, achieving
state-of-the-art results on various benchmarks.
669e0b16de95466572dad018
Ofir Abramovich
claimed_verified
2024-07-22T07:51:16.580Z
669e089bf433fc42beae9923
/avatars/9e06da07e9b9ac0bdee0b28ccabeb0d1.svg
Ofir Abramovich
user
ofirab
669e0b16de95466572dad019
Niv Nayman
669e0b16de95466572dad01a
Sharon Fogel
669e0b16de95466572dad01b
Inbal Lavi
669e0b16de95466572dad01c
Ron Litman
admin_assigned
2024-07-22T10:04:54.175Z
65e70aa92db014a1b278a857
/avatars/2f586a73408de49af10c1efe6abcb9cf.svg
Ron Litman
user
ronlitman
669e0b16de95466572dad01d
Shahar Tsiper
669e0b16de95466572dad01e
Royee Tichauer
admin_assigned
2024-07-22T10:05:49.820Z
64666d68dac127ac80f301c4
/avatars/973848cc702e9bd9fed80b9b2678facf.svg
Royee Tichauer
user
royeet
669e0b16de95466572dad01f
Srikar Appalaraju
669e0b16de95466572dad020
Shai Mazor
669e0b16de95466572dad021
R. Manmatha
VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document
  Understanding
In recent years, notable advancements have been made in the domain of visual
document understanding, with the prevailing architecture comprising a cascade
of vision and language models. The text component can either be extracted
explicitly with the use of external OCR models in OCR-based approaches, or
alternatively, the vision model can be endowed with reading capabilities in
OCR-free approaches. Typically, the queries to the model are input exclusively
to the language component, necessitating the visual features to encompass the
entire document. In this paper, we present VisFocus, an OCR-free method
designed to better exploit the vision encoder's capacity by coupling it
directly with the language prompt. To do so, we replace the down-sampling
layers with layers that receive the input prompt and allow highlighting
relevant parts of the document, while disregarding others. We pair the
architecture enhancements with a novel pre-training task, using language
masking on a snippet of the document text fed to the visual encoder in place of
the prompt, to empower the model with focusing capabilities. Consequently,
VisFocus learns to allocate its attention to text patches pertinent to the
provided prompt. Our experiments demonstrate that this prompt-guided visual
encoding approach significantly improves performance, achieving
state-of-the-art results on various benchmarks.
VisFocus enhances visual document understanding by integrating language prompts directly into the vision encoder, enabling focus on relevant text patches and improving performance across benchmarks.
vision encoder language prompt down-sampling layers language masking prompt-guided visual encoding