title: Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large
    Language Models
summary: Retrieval-Augmented Generation (RAG) has been shown to enhance the factual
  accuracy of Large Language Models (LLMs), but existing methods often suffer
  from limited reasoning capabilities in effectively using the retrieved
  evidence, particularly when using open-source LLMs. To mitigate this gap, we
  introduce a novel framework, Open-RAG, designed to enhance reasoning
  capabilities in RAG with open-source LLMs. Our framework transforms an
  arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE)
  model capable of handling complex reasoning tasks, including both single- and
  multi-hop queries. Open-RAG uniquely trains the model to navigate challenging
  distractors that appear relevant but are misleading. As a result, Open-RAG
  leverages latent learning, dynamically selecting relevant experts and
  integrating external knowledge effectively for more accurate and contextually
  relevant responses. In addition, we propose a hybrid adaptive retrieval method
  to determine retrieval necessity and balance the trade-off between performance
  gain and inference speed. Experimental results show that the Llama2-7B-based
  Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT,
  Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source
  our code and models at https://openragmoe.github.io/
paper_authors._id: 66fe6c0e162623b07c29a302
paper_authors.name: Shayekh Bin Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T10:17:56.369Z
paper_authors._id: 5e4b943a37cb5b49818287b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
paper_authors.fullname: Shayekh Islam
paper_authors.type: user
paper_authors.user: shayekh
paper_authors._id: 66fe6c0e162623b07c29a303
paper_authors.name: Md Asib Rahman
paper_authors._id: 66fe6c0e162623b07c29a304
paper_authors.name: K S M Tozammel Hossain
paper_authors._id: 66fe6c0e162623b07c29a305
paper_authors.name: Enamul Hoque
paper_authors._id: 66fe6c0e162623b07c29a306
paper_authors.name: Shafiq Joty
paper_authors._id: 66fe6c0e162623b07c29a307
paper_authors.name: Md Rizwan Parvez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T10:20:33.642Z
paper_authors._id: 65ae1c4468139e3c42973fe4
paper_authors.avatarUrl: /avatars/b065a857dd763410caadea37a2dc01c4.svg
paper_authors.fullname: Md Rizwan Parvez
paper_authors.type: user
paper_authors.user: mparvez
paper_title: Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large
    Language Models
paper_summary: Retrieval-Augmented Generation (RAG) has been shown to enhance the factual
  accuracy of Large Language Models (LLMs), but existing methods often suffer
  from limited reasoning capabilities in effectively using the retrieved
  evidence, particularly when using open-source LLMs. To mitigate this gap, we
  introduce a novel framework, Open-RAG, designed to enhance reasoning
  capabilities in RAG with open-source LLMs. Our framework transforms an
  arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE)
  model capable of handling complex reasoning tasks, including both single- and
  multi-hop queries. Open-RAG uniquely trains the model to navigate challenging
  distractors that appear relevant but are misleading. As a result, Open-RAG
  leverages latent learning, dynamically selecting relevant experts and
  integrating external knowledge effectively for more accurate and contextually
  relevant responses. In addition, we propose a hybrid adaptive retrieval method
  to determine retrieval necessity and balance the trade-off between performance
  gain and inference speed. Experimental results show that the Llama2-7B-based
  Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT,
  Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source
  our code and models at https://openragmoe.github.io/
paper_ai_summary: Open-RAG enhances open-source LLMs by transforming them into sparse mixture-of-experts models, improving reasoning capabilities and factual accuracy in knowledge-intensive tasks.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) Large Language Models (LLMs) parameter-efficient sparse mixture of experts (MoE) single-hop queries multi-hop queries latent learning hybrid adaptive retrieval method