VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
  Documents
Retrieval-augmented generation (RAG) is an effective technique that enables
large language models (LLMs) to utilize external knowledge sources for
generation. However, current RAG systems are solely based on text, rendering it
impossible to utilize vision information like layout and images that play
crucial roles in real-world multi-modality documents. In this paper, we
introduce VisRAG, which tackles this issue by establishing a vision-language
model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
document to obtain text, the document is directly embedded using a VLM as an
image and then retrieved to enhance the generation of a VLM. Compared to
traditional text-based RAG, VisRAG maximizes the retention and utilization of
the data information in the original documents, eliminating the information
loss introduced during the parsing process. We collect both open-source and
synthetic data to train the retriever in VisRAG and explore a variety of
generation methods. Experiments demonstrate that VisRAG outperforms traditional
RAG in both the retrieval and generation stages, achieving a 25--39\%
end-to-end performance gain over traditional text-based RAG pipeline. Further
analysis reveals that VisRAG is effective in utilizing training data and
demonstrates strong generalization capability, positioning it as a promising
solution for RAG on multi-modality documents. Our code and data are available
at https://github.com/openbmb/visrag .
670e4bb54968ab91d837d73e
Shi Yu
claimed_verified
2024-10-15T16:37:25.335Z
6135eeeb5bc6ecdf86b60f0d
/avatars/43cedcf20ab6b0801a662787400e1384.svg
Shi Yu
user
yushi
670e4bb54968ab91d837d73f
Chaoyue Tang
claimed_verified
2024-10-15T16:37:22.575Z
65e2db6899d809668f2f29f5
/avatars/33365712acb3d8b0fc935accd8086333.svg
Chaoyue Tang
user
tcy6
670e4bb54968ab91d837d740
Bokai Xu
claimed_verified
2024-10-15T16:37:23.899Z
6415818a986557e8cac252bf
https://cdn-avatars.huggingface.co/v1/production/uploads/6415818a986557e8cac252bf/T4u9qjRt8P4clF4nOTA4W.jpeg
Bokai Xu
user
bokesyo
670e4bb54968ab91d837d741
Junbo Cui
claimed_verified
2024-10-15T15:09:53.369Z
63f706dfe94ed998c463ed66
https://cdn-avatars.huggingface.co/v1/production/uploads/63f706dfe94ed998c463ed66/tAGil2qiFNev6CfEEDseV.png
Cuiunbo
user
Cuiunbo
670e4bb54968ab91d837d742
Junhao Ran
670e4bb54968ab91d837d743
Yukun Yan
670e4bb54968ab91d837d744
Zhenghao Liu
670e4bb54968ab91d837d745
Shuo Wang
670e4bb54968ab91d837d746
Xu Han
670e4bb54968ab91d837d747
Zhiyuan Liu
claimed_verified
2025-06-02T07:56:56.592Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
670e4bb54968ab91d837d748
Maosong Sun
VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
  Documents
Retrieval-augmented generation (RAG) is an effective technique that enables
large language models (LLMs) to utilize external knowledge sources for
generation. However, current RAG systems are solely based on text, rendering it
impossible to utilize vision information like layout and images that play
crucial roles in real-world multi-modality documents. In this paper, we
introduce VisRAG, which tackles this issue by establishing a vision-language
model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
document to obtain text, the document is directly embedded using a VLM as an
image and then retrieved to enhance the generation of a VLM. Compared to
traditional text-based RAG, VisRAG maximizes the retention and utilization of
the data information in the original documents, eliminating the information
loss introduced during the parsing process. We collect both open-source and
synthetic data to train the retriever in VisRAG and explore a variety of
generation methods. Experiments demonstrate that VisRAG outperforms traditional
RAG in both the retrieval and generation stages, achieving a 25--39\%
end-to-end performance gain over traditional text-based RAG pipeline. Further
analysis reveals that VisRAG is effective in utilizing training data and
demonstrates strong generalization capability, positioning it as a promising
solution for RAG on multi-modality documents. Our code and data are available
at https://github.com/openbmb/visrag .
VisRAG, a vision-language model-based RAG pipeline, enhances document generation by directly embedding multi-modal documents and outperforms traditional text-based RAG systems.
Retrieval-augmented generation (RAG) large language models (LLMs) vision information vision-language model (VLM) multi-modality documents data embedding information retention performance gain generalization capability