AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
  Retrieval-Augmented Generation
Recent advancements in Large Language Models have transformed ML/AI
development, necessitating a reevaluation of AutoML principles for the
Retrieval-Augmented Generation (RAG) systems. To address the challenges of
hyper-parameter optimization and online adaptation in RAG, we propose the
AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
MAB (Hier-MAB) method for efficient exploration of large search spaces. We
conduct extensive experiments on tuning hyper-parameters, such as top-k
retrieved documents, prompt compression ratio, and embedding methods, using the
ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
optimization all three hyper-parameters demonstrate that MAB-based online
learning methods can achieve Recall@5 approx 0.8 for scenarios with
prominent gradients in search space, using only sim20% of the LLM API calls
required by the Grid Search approach. Additionally, the proposed Hier-MAB
approach outperforms other baselines in more challenging optimization
scenarios. The code will be made available at https://aka.ms/autorag.
668234bb7ad6fe37b198f6a8
Jia Fu
668234bb7ad6fe37b198f6a9
Xiaoting Qin
admin_assigned
2024-07-01T09:14:16.094Z
662b0bc9c709a61df8291c0f
/avatars/16dd4d945e9fbef5ac889a8087101ded.svg
Xiaoting Qin
user
XiaotingQin
668234bb7ad6fe37b198f6aa
Fangkai Yang
claimed_verified
2025-02-27T09:18:38.630Z
669dcf6200970c3b27aafa5d
/avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
kaikai yang
user
keanudicap
668234bb7ad6fe37b198f6ab
Lu Wang
668234bb7ad6fe37b198f6ac
Jue Zhang
admin_assigned
2024-07-01T09:10:40.195Z
65f40e43653c231cbaf7d1e4
/avatars/a42ac5454cbe175f04c3420fce90cad2.svg
Jue Zhang
user
JueZhang
668234bb7ad6fe37b198f6ad
Qingwei Lin
668234bb7ad6fe37b198f6ae
Yubo Chen
admin_assigned
2024-07-01T09:14:47.607Z
64423a910f373bdb0463bf25
/avatars/6b41c4eb817836d50d0072009f124f29.svg
Yubo Chen
user
ybch14
668234bb7ad6fe37b198f6af
Dongmei Zhang
668234bb7ad6fe37b198f6b0
Saravan Rajmohan
668234bb7ad6fe37b198f6b1
Qi Zhang
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
  Retrieval-Augmented Generation
Recent advancements in Large Language Models have transformed ML/AI
development, necessitating a reevaluation of AutoML principles for the
Retrieval-Augmented Generation (RAG) systems. To address the challenges of
hyper-parameter optimization and online adaptation in RAG, we propose the
AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
MAB (Hier-MAB) method for efficient exploration of large search spaces. We
conduct extensive experiments on tuning hyper-parameters, such as top-k
retrieved documents, prompt compression ratio, and embedding methods, using the
ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
optimization all three hyper-parameters demonstrate that MAB-based online
learning methods can achieve Recall@5 approx 0.8 for scenarios with
prominent gradients in search space, using only sim20% of the LLM API calls
required by the Grid Search approach. Additionally, the proposed Hier-MAB
approach outperforms other baselines in more challenging optimization
scenarios. The code will be made available at https://aka.ms/autorag.
AutoRAG-HP uses a Hierarchical MAB method to efficiently optimize hyperparameters in RAG systems, achieving better recall with reduced API calls compared to grid search.
Large Language Models AutoML Retrieval-Augmented Generation (RAG) hyper-parameter optimization online adaptation multi-armed bandit (MAB) Hierarchical MAB (Hier-MAB) recall@5 ALCE-ASQA Natural Questions grid search