Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large
  Language Models
Retrieval-Augmented Generation (RAG) has been shown to enhance the factual
accuracy of Large Language Models (LLMs), but existing methods often suffer
from limited reasoning capabilities in effectively using the retrieved
evidence, particularly when using open-source LLMs. To mitigate this gap, we
introduce a novel framework, Open-RAG, designed to enhance reasoning
capabilities in RAG with open-source LLMs. Our framework transforms an
arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE)
model capable of handling complex reasoning tasks, including both single- and
multi-hop queries. Open-RAG uniquely trains the model to navigate challenging
distractors that appear relevant but are misleading. As a result, Open-RAG
leverages latent learning, dynamically selecting relevant experts and
integrating external knowledge effectively for more accurate and contextually
relevant responses. In addition, we propose a hybrid adaptive retrieval method
to determine retrieval necessity and balance the trade-off between performance
gain and inference speed. Experimental results show that the Llama2-7B-based
Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT,
Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source
our code and models at https://openragmoe.github.io/
66fe6c0e162623b07c29a302
Shayekh Bin Islam
claimed_verified
2024-10-03T10:17:56.369Z
5e4b943a37cb5b49818287b5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
Shayekh Islam
user
shayekh
66fe6c0e162623b07c29a303
Md Asib Rahman
66fe6c0e162623b07c29a304
K S M Tozammel Hossain
66fe6c0e162623b07c29a305
Enamul Hoque
66fe6c0e162623b07c29a306
Shafiq Joty
66fe6c0e162623b07c29a307
Md Rizwan Parvez
claimed_verified
2024-12-05T10:20:33.642Z
65ae1c4468139e3c42973fe4
/avatars/b065a857dd763410caadea37a2dc01c4.svg
Md Rizwan Parvez
user
mparvez
Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large
  Language Models
Retrieval-Augmented Generation (RAG) has been shown to enhance the factual
accuracy of Large Language Models (LLMs), but existing methods often suffer
from limited reasoning capabilities in effectively using the retrieved
evidence, particularly when using open-source LLMs. To mitigate this gap, we
introduce a novel framework, Open-RAG, designed to enhance reasoning
capabilities in RAG with open-source LLMs. Our framework transforms an
arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE)
model capable of handling complex reasoning tasks, including both single- and
multi-hop queries. Open-RAG uniquely trains the model to navigate challenging
distractors that appear relevant but are misleading. As a result, Open-RAG
leverages latent learning, dynamically selecting relevant experts and
integrating external knowledge effectively for more accurate and contextually
relevant responses. In addition, we propose a hybrid adaptive retrieval method
to determine retrieval necessity and balance the trade-off between performance
gain and inference speed. Experimental results show that the Llama2-7B-based
Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT,
Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source
our code and models at https://openragmoe.github.io/
Open-RAG enhances open-source LLMs by transforming them into sparse mixture-of-experts models, improving reasoning capabilities and factual accuracy in knowledge-intensive tasks.
Retrieval-Augmented Generation (RAG) Large Language Models (LLMs) parameter-efficient sparse mixture of experts (MoE) single-hop queries multi-hop queries latent learning hybrid adaptive retrieval method