title: ColPali: Efficient Document Retrieval with Vision Language Models
summary: Documents are visually rich structures that convey information through text,
  as well as tables, figures, page layouts, or fonts. While modern document
  retrieval systems exhibit strong performance on query-to-text matching, they
  struggle to exploit visual cues efficiently, hindering their performance on
  practical document retrieval applications such as Retrieval Augmented
  Generation. To benchmark current systems on visually rich document retrieval,
  we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of
  various page-level retrieving tasks spanning multiple domains, languages, and
  settings. The inherent shortcomings of modern systems motivate the introduction
  of a new retrieval model architecture, ColPali, which leverages the document
  understanding capabilities of recent Vision Language Models to produce
  high-quality contextualized embeddings solely from images of document pages.
  Combined with a late interaction matching mechanism, ColPali largely
  outperforms modern document retrieval pipelines while being drastically faster
  and end-to-end trainable.
paper_authors._id: 668397082235ae231ac5eeba
paper_authors.name: Manuel Faysse
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-02T06:09:46.709Z
paper_authors._id: 60f2e021adf471cbdf8bb660
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
paper_authors.fullname: Manuel Faysse
paper_authors.type: user
paper_authors.user: manu
paper_authors._id: 668397082235ae231ac5eebb
paper_authors.name: Hugues Sibille
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T11:59:52.839Z
paper_authors._id: 66211794ae2f58da4f00d317
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66211794ae2f58da4f00d317/kL3nsgW9ri56J0PaaSMkD.jpeg
paper_authors.fullname: Hugues Sibille
paper_authors.type: user
paper_authors.user: HugSib
paper_authors._id: 668397082235ae231ac5eebc
paper_authors.name: Tony Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T08:01:10.759Z
paper_authors._id: 6264f9655f6f2e14d6ac981c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650784534234-noauth.png
paper_authors.fullname: Tony Wu
paper_authors.type: user
paper_authors.user: tonywu71
paper_authors._id: 668397082235ae231ac5eebd
paper_authors.name: Gautier Viaud
paper_authors._id: 668397082235ae231ac5eebe
paper_authors.name: Céline Hudelot
paper_authors._id: 668397082235ae231ac5eebf
paper_authors.name: Pierre Colombo
paper_title: ColPali: Efficient Document Retrieval with Vision Language Models
paper_summary: Documents are visually rich structures that convey information through text,
  as well as tables, figures, page layouts, or fonts. While modern document
  retrieval systems exhibit strong performance on query-to-text matching, they
  struggle to exploit visual cues efficiently, hindering their performance on
  practical document retrieval applications such as Retrieval Augmented
  Generation. To benchmark current systems on visually rich document retrieval,
  we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of
  various page-level retrieving tasks spanning multiple domains, languages, and
  settings. The inherent shortcomings of modern systems motivate the introduction
  of a new retrieval model architecture, ColPali, which leverages the document
  understanding capabilities of recent Vision Language Models to produce
  high-quality contextualized embeddings solely from images of document pages.
  Combined with a late interaction matching mechanism, ColPali largely
  outperforms modern document retrieval pipelines while being drastically faster
  and end-to-end trainable.
paper_ai_summary: The ColPali retrieval model uses Vision Language Models to create embeddings from images of document pages, improving performance and speed in visually rich document retrieval tasks.
paper_ai_keywords: Visual Document Retrieval Benchmark ViDoRe ColPali Vision Language Models contextualized embeddings late interaction matching mechanism