title: VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
    Documents
summary: Retrieval-augmented generation (RAG) is an effective technique that enables
  large language models (LLMs) to utilize external knowledge sources for
  generation. However, current RAG systems are solely based on text, rendering it
  impossible to utilize vision information like layout and images that play
  crucial roles in real-world multi-modality documents. In this paper, we
  introduce VisRAG, which tackles this issue by establishing a vision-language
  model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
  document to obtain text, the document is directly embedded using a VLM as an
  image and then retrieved to enhance the generation of a VLM. Compared to
  traditional text-based RAG, VisRAG maximizes the retention and utilization of
  the data information in the original documents, eliminating the information
  loss introduced during the parsing process. We collect both open-source and
  synthetic data to train the retriever in VisRAG and explore a variety of
  generation methods. Experiments demonstrate that VisRAG outperforms traditional
  RAG in both the retrieval and generation stages, achieving a 25--39\%
  end-to-end performance gain over traditional text-based RAG pipeline. Further
  analysis reveals that VisRAG is effective in utilizing training data and
  demonstrates strong generalization capability, positioning it as a promising
  solution for RAG on multi-modality documents. Our code and data are available
  at https://github.com/openbmb/visrag .
paper_authors._id: 670e4bb54968ab91d837d73e
paper_authors.name: Shi Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T16:37:25.335Z
paper_authors._id: 6135eeeb5bc6ecdf86b60f0d
paper_authors.avatarUrl: /avatars/43cedcf20ab6b0801a662787400e1384.svg
paper_authors.fullname: Shi Yu
paper_authors.type: user
paper_authors.user: yushi
paper_authors._id: 670e4bb54968ab91d837d73f
paper_authors.name: Chaoyue Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T16:37:22.575Z
paper_authors._id: 65e2db6899d809668f2f29f5
paper_authors.avatarUrl: /avatars/33365712acb3d8b0fc935accd8086333.svg
paper_authors.fullname: Chaoyue Tang
paper_authors.type: user
paper_authors.user: tcy6
paper_authors._id: 670e4bb54968ab91d837d740
paper_authors.name: Bokai Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T16:37:23.899Z
paper_authors._id: 6415818a986557e8cac252bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6415818a986557e8cac252bf/T4u9qjRt8P4clF4nOTA4W.jpeg
paper_authors.fullname: Bokai Xu
paper_authors.type: user
paper_authors.user: bokesyo
paper_authors._id: 670e4bb54968ab91d837d741
paper_authors.name: Junbo Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T15:09:53.369Z
paper_authors._id: 63f706dfe94ed998c463ed66
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f706dfe94ed998c463ed66/tAGil2qiFNev6CfEEDseV.png
paper_authors.fullname: Cuiunbo
paper_authors.type: user
paper_authors.user: Cuiunbo
paper_authors._id: 670e4bb54968ab91d837d742
paper_authors.name: Junhao Ran
paper_authors._id: 670e4bb54968ab91d837d743
paper_authors.name: Yukun Yan
paper_authors._id: 670e4bb54968ab91d837d744
paper_authors.name: Zhenghao Liu
paper_authors._id: 670e4bb54968ab91d837d745
paper_authors.name: Shuo Wang
paper_authors._id: 670e4bb54968ab91d837d746
paper_authors.name: Xu Han
paper_authors._id: 670e4bb54968ab91d837d747
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:56:56.592Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 670e4bb54968ab91d837d748
paper_authors.name: Maosong Sun
paper_title: VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
    Documents
paper_summary: Retrieval-augmented generation (RAG) is an effective technique that enables
  large language models (LLMs) to utilize external knowledge sources for
  generation. However, current RAG systems are solely based on text, rendering it
  impossible to utilize vision information like layout and images that play
  crucial roles in real-world multi-modality documents. In this paper, we
  introduce VisRAG, which tackles this issue by establishing a vision-language
  model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
  document to obtain text, the document is directly embedded using a VLM as an
  image and then retrieved to enhance the generation of a VLM. Compared to
  traditional text-based RAG, VisRAG maximizes the retention and utilization of
  the data information in the original documents, eliminating the information
  loss introduced during the parsing process. We collect both open-source and
  synthetic data to train the retriever in VisRAG and explore a variety of
  generation methods. Experiments demonstrate that VisRAG outperforms traditional
  RAG in both the retrieval and generation stages, achieving a 25--39\%
  end-to-end performance gain over traditional text-based RAG pipeline. Further
  analysis reveals that VisRAG is effective in utilizing training data and
  demonstrates strong generalization capability, positioning it as a promising
  solution for RAG on multi-modality documents. Our code and data are available
  at https://github.com/openbmb/visrag .
paper_ai_summary: VisRAG, a vision-language model-based RAG pipeline, enhances document generation by directly embedding multi-modal documents and outperforms traditional text-based RAG systems.
paper_ai_keywords: Retrieval-augmented generation (RAG) large language models (LLMs) vision information vision-language model (VLM) multi-modality documents data embedding information retention performance gain generalization capability