title: UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with
    Diverse Modalities and Granularities
summary: Retrieval-Augmented Generation (RAG) has shown substantial promise in
  improving factual accuracy by grounding model responses with external knowledge
  relevant to queries. However, most existing RAG approaches are limited to a
  text-only corpus, and while recent efforts have extended RAG to other
  modalities such as images and videos, they typically operate over a single
  modality-specific corpus. In contrast, real-world queries vary widely in the
  type of knowledge they require, which a single type of knowledge source cannot
  address. To address this, we introduce UniversalRAG, a novel RAG framework
  designed to retrieve and integrate knowledge from heterogeneous sources with
  diverse modalities and granularities. Specifically, motivated by the
  observation that forcing all modalities into a unified representation space
  derived from a single combined corpus causes a modality gap, where the
  retrieval tends to favor items from the same modality as the query, we propose
  a modality-aware routing mechanism that dynamically identifies the most
  appropriate modality-specific corpus and performs targeted retrieval within it.
  Also, beyond modality, we organize each modality into multiple granularity
  levels, enabling fine-tuned retrieval tailored to the complexity and scope of
  the query. We validate UniversalRAG on 8 benchmarks spanning multiple
  modalities, showing its superiority over modality-specific and unified
  baselines.
paper_authors._id: 6811966ae20ba7d0683b8adc
paper_authors.name: Woongyeong Yeo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:03.853Z
paper_authors._id: 66d30f5fad293ffc4b7672bc
paper_authors.avatarUrl: /avatars/6f164d813b947940a088820f8fd4dbe8.svg
paper_authors.fullname: Woongyeong Yeo
paper_authors.type: user
paper_authors.user: wgcyeo
paper_authors._id: 6811966ae20ba7d0683b8add
paper_authors.name: Kangsan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:00.948Z
paper_authors._id: 66ed7737f2f27a5dfd81ef09
paper_authors.avatarUrl: /avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
paper_authors.fullname: Kangsan Kim
paper_authors.type: user
paper_authors.user: KangsanKim71
paper_authors._id: 6811966ae20ba7d0683b8ade
paper_authors.name: Soyeong Jeong
paper_authors._id: 6811966ae20ba7d0683b8adf
paper_authors.name: Jinheon Baek
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:07.351Z
paper_authors._id: 63036b6c5c70c21d0ea79d48
paper_authors.avatarUrl: /avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
paper_authors.fullname: Jinheon Baek
paper_authors.type: user
paper_authors.user: jinheon
paper_authors._id: 6811966ae20ba7d0683b8ae0
paper_authors.name: Sung Ju Hwang
paper_title: UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with
    Diverse Modalities and Granularities
paper_summary: Retrieval-Augmented Generation (RAG) has shown substantial promise in
  improving factual accuracy by grounding model responses with external knowledge
  relevant to queries. However, most existing RAG approaches are limited to a
  text-only corpus, and while recent efforts have extended RAG to other
  modalities such as images and videos, they typically operate over a single
  modality-specific corpus. In contrast, real-world queries vary widely in the
  type of knowledge they require, which a single type of knowledge source cannot
  address. To address this, we introduce UniversalRAG, a novel RAG framework
  designed to retrieve and integrate knowledge from heterogeneous sources with
  diverse modalities and granularities. Specifically, motivated by the
  observation that forcing all modalities into a unified representation space
  derived from a single combined corpus causes a modality gap, where the
  retrieval tends to favor items from the same modality as the query, we propose
  a modality-aware routing mechanism that dynamically identifies the most
  appropriate modality-specific corpus and performs targeted retrieval within it.
  Also, beyond modality, we organize each modality into multiple granularity
  levels, enabling fine-tuned retrieval tailored to the complexity and scope of
  the query. We validate UniversalRAG on 8 benchmarks spanning multiple
  modalities, showing its superiority over modality-specific and unified
  baselines.
paper_ai_summary: UniversalRAG enhances Retrieval-Augmented Generation by dynamically selecting and retrieving knowledge from various heterogeneous and modality-specific sources to improve factual accuracy across different types of queries.
paper_ai_keywords: Retrieval-Augmented Generation RAG factual accuracy external knowledge model responses queries modalities images videos UniversalRAG modality-aware routing mechanism granularities modality gap retrieval baselines