title: Unifying Multimodal Retrieval via Document Screenshot Embedding
summary: In the real world, documents are organized in different formats and varied
  modalities. Traditional retrieval pipelines require tailored document parsing
  techniques and content extraction modules to prepare input for indexing. This
  process is tedious, prone to errors, and has information loss. To this end, we
  propose Document Screenshot Embedding} (DSE), a novel retrieval paradigm that
  regards document screenshots as a unified input format, which does not require
  any content extraction preprocess and preserves all the information in a
  document (e.g., text, image and layout). DSE leverages a large vision-language
  model to directly encode document screenshots into dense representations for
  retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a
  1.3M Wikipedia web page screenshots as the corpus to answer the questions from
  the Natural Questions dataset. In such a text-intensive document retrieval
  setting, DSE shows competitive effectiveness compared to other text retrieval
  methods relying on parsing. For example, DSE outperforms BM25 by 17 points in
  top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide
  retrieval, DSE significantly outperforms OCR text retrieval methods by over 15
  points in nDCG@10. These experiments show that DSE is an effective document
  retrieval paradigm for diverse types of documents. Model checkpoints, code, and
  Wiki-SS collection will be released.
paper_authors._id: 6671cf08cbf550c42ac91b45
paper_authors.name: Xueguang Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:26:49.058Z
paper_authors._id: 5ec82854968f6028e0559f70
paper_authors.avatarUrl: /avatars/45b58d912f7d00cb351947cd79d5eeb4.svg
paper_authors.fullname: Xueguang Ma
paper_authors.type: user
paper_authors.user: MrLight
paper_authors._id: 6671cf08cbf550c42ac91b46
paper_authors.name: Sheng-Chieh Lin
paper_authors._id: 6671cf08cbf550c42ac91b47
paper_authors.name: Minghan Li
paper_authors._id: 6671cf08cbf550c42ac91b48
paper_authors.name: Wenhu Chen
paper_authors._id: 6671cf08cbf550c42ac91b49
paper_authors.name: Jimmy Lin
paper_title: Unifying Multimodal Retrieval via Document Screenshot Embedding
paper_summary: In the real world, documents are organized in different formats and varied
  modalities. Traditional retrieval pipelines require tailored document parsing
  techniques and content extraction modules to prepare input for indexing. This
  process is tedious, prone to errors, and has information loss. To this end, we
  propose Document Screenshot Embedding} (DSE), a novel retrieval paradigm that
  regards document screenshots as a unified input format, which does not require
  any content extraction preprocess and preserves all the information in a
  document (e.g., text, image and layout). DSE leverages a large vision-language
  model to directly encode document screenshots into dense representations for
  retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a
  1.3M Wikipedia web page screenshots as the corpus to answer the questions from
  the Natural Questions dataset. In such a text-intensive document retrieval
  setting, DSE shows competitive effectiveness compared to other text retrieval
  methods relying on parsing. For example, DSE outperforms BM25 by 17 points in
  top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide
  retrieval, DSE significantly outperforms OCR text retrieval methods by over 15
  points in nDCG@10. These experiments show that DSE is an effective document
  retrieval paradigm for diverse types of documents. Model checkpoints, code, and
  Wiki-SS collection will be released.
paper_ai_summary: Document Screenshot Embedding leverages a vision-language model to directly encode document screenshots for retrieval, preserving all information and competing effectively with text-based methods.
paper_ai_keywords: Document Screenshot Embedding DSE vision-language model dense representations retrieval Wiki-SS Natural Questions BM25 nDCG@10 OCR text retrieval