title: VideoRAG: Retrieval-Augmented Generation over Video Corpus
summary: Retrieval-Augmented Generation (RAG) is a powerful strategy to address the
  issue of generating factually incorrect outputs in foundation models by
  retrieving external knowledge relevant to queries and incorporating it into
  their generation process. However, existing RAG approaches have primarily
  focused on textual information, with some recent advancements beginning to
  consider images, and they largely overlook videos, a rich source of multimodal
  knowledge capable of representing events, processes, and contextual details
  more effectively than any other modality. While a few recent studies explore
  the integration of videos in the response generation process, they either
  predefine query-associated videos without retrieving them according to queries,
  or convert videos into the textual descriptions without harnessing their
  multimodal richness. To tackle these, we introduce VideoRAG, a novel framework
  that not only dynamically retrieves relevant videos based on their relevance
  with queries but also utilizes both visual and textual information of videos in
  the output generation. Further, to operationalize this, our method revolves
  around the recent advance of Large Video Language Models (LVLMs), which enable
  the direct processing of video content to represent it for retrieval and
  seamless integration of the retrieved videos jointly with queries. We
  experimentally validate the effectiveness of VideoRAG, showcasing that it is
  superior to relevant baselines.
paper_authors._id: 678474cb65495e1686093192
paper_authors.name: Soyeong Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T09:18:35.528Z
paper_authors._id: 64e5a1cd4c20016ec9020ec8
paper_authors.avatarUrl: /avatars/d7ffe7fbbe39c0a013375357457c57b3.svg
paper_authors.fullname: Soyeong
paper_authors.type: user
paper_authors.user: starsuzi
paper_authors._id: 678474cb65495e1686093193
paper_authors.name: Kangsan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:32.431Z
paper_authors._id: 66ed7737f2f27a5dfd81ef09
paper_authors.avatarUrl: /avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
paper_authors.fullname: Kangsan Kim
paper_authors.type: user
paper_authors.user: KangsanKim71
paper_authors._id: 678474cb65495e1686093194
paper_authors.name: Jinheon Baek
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:15:25.765Z
paper_authors._id: 63036b6c5c70c21d0ea79d48
paper_authors.avatarUrl: /avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
paper_authors.fullname: Jinheon Baek
paper_authors.type: user
paper_authors.user: jinheon
paper_authors._id: 678474cb65495e1686093195
paper_authors.name: Sung Ju Hwang
paper_title: VideoRAG: Retrieval-Augmented Generation over Video Corpus
paper_summary: Retrieval-Augmented Generation (RAG) is a powerful strategy to address the
  issue of generating factually incorrect outputs in foundation models by
  retrieving external knowledge relevant to queries and incorporating it into
  their generation process. However, existing RAG approaches have primarily
  focused on textual information, with some recent advancements beginning to
  consider images, and they largely overlook videos, a rich source of multimodal
  knowledge capable of representing events, processes, and contextual details
  more effectively than any other modality. While a few recent studies explore
  the integration of videos in the response generation process, they either
  predefine query-associated videos without retrieving them according to queries,
  or convert videos into the textual descriptions without harnessing their
  multimodal richness. To tackle these, we introduce VideoRAG, a novel framework
  that not only dynamically retrieves relevant videos based on their relevance
  with queries but also utilizes both visual and textual information of videos in
  the output generation. Further, to operationalize this, our method revolves
  around the recent advance of Large Video Language Models (LVLMs), which enable
  the direct processing of video content to represent it for retrieval and
  seamless integration of the retrieved videos jointly with queries. We
  experimentally validate the effectiveness of VideoRAG, showcasing that it is
  superior to relevant baselines.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) factually incorrect outputs foundation models external knowledge queries textual information images videos multimodal knowledge events processes contextual details VideoRAG dynamically retrieves visual information textual information output generation Large Video Language Models (LVLMs) direct processing video content retrieval and seamless integration baselines