title: AutoMix: Automatically Mixing Language Models
summary: Large language models (LLMs) are now available in various sizes and
  configurations from cloud API providers. While this diversity offers a broad
  spectrum of choices, effectively leveraging the options to optimize
  computational cost and performance remains challenging. In this work, we
  present AutoMix, an approach that strategically routes queries to larger LMs,
  based on the approximate correctness of outputs from a smaller LM. Central to
  AutoMix is a few-shot self-verification mechanism, which estimates the
  reliability of its own outputs without requiring training. Given that
  verifications can be noisy, we employ a meta verifier in AutoMix to refine the
  accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five
  context-grounded reasoning datasets demonstrate that AutoMix surpasses
  established baselines, improving the incremental benefit per cost by up to 89%.
  Our code and data are available at https://github.com/automix-llm/automix.
paper_authors._id: 65320eaec8da3465f480d99e
paper_authors.name: Aman Madaan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:40:09.589Z
paper_authors._id: 61bbc2b2d96f5d4111901dcb
paper_authors.avatarUrl: /avatars/83c826d5cad7bab27d33585cb7e64575.svg
paper_authors.fullname: Aman Madaan
paper_authors.type: user
paper_authors.user: amanmadaan
paper_authors._id: 65320eaec8da3465f480d99f
paper_authors.name: Pranjal Aggarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:40:32.283Z
paper_authors._id: 60f3280b702e4ecabac96740
paper_authors.avatarUrl: /avatars/ce12e328352aac25f0370333bf7f6cce.svg
paper_authors.fullname: Pranjal Aggarwal
paper_authors.type: user
paper_authors.user: Pranjal2041
paper_authors._id: 65320eaec8da3465f480d9a0
paper_authors.name: Ankit Anand
paper_authors._id: 65320eaec8da3465f480d9a1
paper_authors.name: Srividya Pranavi Potharaju
paper_authors._id: 65320eaec8da3465f480d9a2
paper_authors.name: Swaroop Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:41:36.585Z
paper_authors._id: 61841feb7c380c66d40cad54
paper_authors.avatarUrl: /avatars/7a79362a24b844860781fb0625527eae.svg
paper_authors.fullname: Swaroop Mishra
paper_authors.type: user
paper_authors.user: swaroop7
paper_authors._id: 65320eaec8da3465f480d9a3
paper_authors.name: Pei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:41:44.573Z
paper_authors._id: 60844fd056f7623a0b84914b
paper_authors.avatarUrl: /avatars/f7d2763258b8c614783ead9480fbcf8d.svg
paper_authors.fullname: Pei Zhou
paper_authors.type: user
paper_authors.user: peiz
paper_authors._id: 65320eaec8da3465f480d9a4
paper_authors.name: Aditya Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-21T09:22:35.175Z
paper_authors._id: 64bac8eb01f1983a862e58b2
paper_authors.avatarUrl: /avatars/ba005db511bf9fa2e79ca2e78852f8fd.svg
paper_authors.fullname: Aditya Gupta
paper_authors.type: user
paper_authors.user: aditya2211
paper_authors._id: 65320eaec8da3465f480d9a5
paper_authors.name: Dheeraj Rajagopal
paper_authors._id: 65320eaec8da3465f480d9a6
paper_authors.name: Karthik Kappaganthu
paper_authors._id: 65320eaec8da3465f480d9a7
paper_authors.name: Yiming Yang
paper_authors._id: 65320eaec8da3465f480d9a8
paper_authors.name: Shyam Upadhyay
paper_authors._id: 65320eaec8da3465f480d9a9
paper_authors.name: Mausam
paper_authors._id: 65320eaec8da3465f480d9aa
paper_authors.name: Manaal Faruqui
paper_title: AutoMix: Automatically Mixing Language Models
paper_summary: Large language models (LLMs) are now available in various sizes and
  configurations from cloud API providers. While this diversity offers a broad
  spectrum of choices, effectively leveraging the options to optimize
  computational cost and performance remains challenging. In this work, we
  present AutoMix, an approach that strategically routes queries to larger LMs,
  based on the approximate correctness of outputs from a smaller LM. Central to
  AutoMix is a few-shot self-verification mechanism, which estimates the
  reliability of its own outputs without requiring training. Given that
  verifications can be noisy, we employ a meta verifier in AutoMix to refine the
  accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five
  context-grounded reasoning datasets demonstrate that AutoMix surpasses
  established baselines, improving the incremental benefit per cost by up to 89%.
  Our code and data are available at https://github.com/automix-llm/automix.
paper_ai_summary: AutoMix strategically routes queries to larger LLMs based on a noise-refined self-verification mechanism, improving the incremental benefit per cost.
paper_ai_keywords: AutoMix few-shot self-verification meta verifier LLMs large language models context-grounded reasoning LLAMA2-13/70B