title: AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
    Retrieval-Augmented Generation
summary: Recent advancements in Large Language Models have transformed ML/AI
  development, necessitating a reevaluation of AutoML principles for the
  Retrieval-Augmented Generation (RAG) systems. To address the challenges of
  hyper-parameter optimization and online adaptation in RAG, we propose the
  AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
  multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
  MAB (Hier-MAB) method for efficient exploration of large search spaces. We
  conduct extensive experiments on tuning hyper-parameters, such as top-k
  retrieved documents, prompt compression ratio, and embedding methods, using the
  ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
  optimization all three hyper-parameters demonstrate that MAB-based online
  learning methods can achieve Recall@5 approx 0.8 for scenarios with
  prominent gradients in search space, using only sim20% of the LLM API calls
  required by the Grid Search approach. Additionally, the proposed Hier-MAB
  approach outperforms other baselines in more challenging optimization
  scenarios. The code will be made available at https://aka.ms/autorag.
paper_authors._id: 668234bb7ad6fe37b198f6a8
paper_authors.name: Jia Fu
paper_authors._id: 668234bb7ad6fe37b198f6a9
paper_authors.name: Xiaoting Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:14:16.094Z
paper_authors._id: 662b0bc9c709a61df8291c0f
paper_authors.avatarUrl: /avatars/16dd4d945e9fbef5ac889a8087101ded.svg
paper_authors.fullname: Xiaoting Qin
paper_authors.type: user
paper_authors.user: XiaotingQin
paper_authors._id: 668234bb7ad6fe37b198f6aa
paper_authors.name: Fangkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:18:38.630Z
paper_authors._id: 669dcf6200970c3b27aafa5d
paper_authors.avatarUrl: /avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
paper_authors.fullname: kaikai yang
paper_authors.type: user
paper_authors.user: keanudicap
paper_authors._id: 668234bb7ad6fe37b198f6ab
paper_authors.name: Lu Wang
paper_authors._id: 668234bb7ad6fe37b198f6ac
paper_authors.name: Jue Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:10:40.195Z
paper_authors._id: 65f40e43653c231cbaf7d1e4
paper_authors.avatarUrl: /avatars/a42ac5454cbe175f04c3420fce90cad2.svg
paper_authors.fullname: Jue Zhang
paper_authors.type: user
paper_authors.user: JueZhang
paper_authors._id: 668234bb7ad6fe37b198f6ad
paper_authors.name: Qingwei Lin
paper_authors._id: 668234bb7ad6fe37b198f6ae
paper_authors.name: Yubo Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:14:47.607Z
paper_authors._id: 64423a910f373bdb0463bf25
paper_authors.avatarUrl: /avatars/6b41c4eb817836d50d0072009f124f29.svg
paper_authors.fullname: Yubo Chen
paper_authors.type: user
paper_authors.user: ybch14
paper_authors._id: 668234bb7ad6fe37b198f6af
paper_authors.name: Dongmei Zhang
paper_authors._id: 668234bb7ad6fe37b198f6b0
paper_authors.name: Saravan Rajmohan
paper_authors._id: 668234bb7ad6fe37b198f6b1
paper_authors.name: Qi Zhang
paper_title: AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
    Retrieval-Augmented Generation
paper_summary: Recent advancements in Large Language Models have transformed ML/AI
  development, necessitating a reevaluation of AutoML principles for the
  Retrieval-Augmented Generation (RAG) systems. To address the challenges of
  hyper-parameter optimization and online adaptation in RAG, we propose the
  AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
  multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
  MAB (Hier-MAB) method for efficient exploration of large search spaces. We
  conduct extensive experiments on tuning hyper-parameters, such as top-k
  retrieved documents, prompt compression ratio, and embedding methods, using the
  ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
  optimization all three hyper-parameters demonstrate that MAB-based online
  learning methods can achieve Recall@5 approx 0.8 for scenarios with
  prominent gradients in search space, using only sim20% of the LLM API calls
  required by the Grid Search approach. Additionally, the proposed Hier-MAB
  approach outperforms other baselines in more challenging optimization
  scenarios. The code will be made available at https://aka.ms/autorag.
paper_ai_keywords: Large Language Models (LLM) AutoML principles Retrieval-Augmented Generation (RAG) hyper-parameter optimization online adaptations AutoRAG-HP framework multi-armed bandit (MAB) Hierarchical MAB (Hier-MAB) top-k retrieved documents prompt compression ratio embedding methods ALCE-ASQA dataset Natural Questions dataset Recall@5 LLM API calls Grid Search