AutoMix: Automatically Mixing Language Models
Large language models (LLMs) are now available in various sizes and
configurations from cloud API providers. While this diversity offers a broad
spectrum of choices, effectively leveraging the options to optimize
computational cost and performance remains challenging. In this work, we
present AutoMix, an approach that strategically routes queries to larger LMs,
based on the approximate correctness of outputs from a smaller LM. Central to
AutoMix is a few-shot self-verification mechanism, which estimates the
reliability of its own outputs without requiring training. Given that
verifications can be noisy, we employ a meta verifier in AutoMix to refine the
accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five
context-grounded reasoning datasets demonstrate that AutoMix surpasses
established baselines, improving the incremental benefit per cost by up to 89%.
Our code and data are available at https://github.com/automix-llm/automix.
65320eaec8da3465f480d99e
Aman Madaan
admin_assigned
2023-10-20T08:40:09.589Z
61bbc2b2d96f5d4111901dcb
/avatars/83c826d5cad7bab27d33585cb7e64575.svg
Aman Madaan
user
amanmadaan
65320eaec8da3465f480d99f
Pranjal Aggarwal
admin_assigned
2023-10-20T08:40:32.283Z
60f3280b702e4ecabac96740
/avatars/ce12e328352aac25f0370333bf7f6cce.svg
Pranjal Aggarwal
user
Pranjal2041
65320eaec8da3465f480d9a0
Ankit Anand
65320eaec8da3465f480d9a1
Srividya Pranavi Potharaju
65320eaec8da3465f480d9a2
Swaroop Mishra
admin_assigned
2023-10-20T08:41:36.585Z
61841feb7c380c66d40cad54
/avatars/7a79362a24b844860781fb0625527eae.svg
Swaroop Mishra
user
swaroop7
65320eaec8da3465f480d9a3
Pei Zhou
admin_assigned
2023-10-20T08:41:44.573Z
60844fd056f7623a0b84914b
/avatars/f7d2763258b8c614783ead9480fbcf8d.svg
Pei Zhou
user
peiz
65320eaec8da3465f480d9a4
Aditya Gupta
claimed_verified
2023-10-21T09:22:35.175Z
64bac8eb01f1983a862e58b2
/avatars/ba005db511bf9fa2e79ca2e78852f8fd.svg
Aditya Gupta
user
aditya2211
65320eaec8da3465f480d9a5
Dheeraj Rajagopal
65320eaec8da3465f480d9a6
Karthik Kappaganthu
65320eaec8da3465f480d9a7
Yiming Yang
65320eaec8da3465f480d9a8
Shyam Upadhyay
65320eaec8da3465f480d9a9
Mausam
65320eaec8da3465f480d9aa
Manaal Faruqui
AutoMix: Automatically Mixing Language Models
Large language models (LLMs) are now available in various sizes and
configurations from cloud API providers. While this diversity offers a broad
spectrum of choices, effectively leveraging the options to optimize
computational cost and performance remains challenging. In this work, we
present AutoMix, an approach that strategically routes queries to larger LMs,
based on the approximate correctness of outputs from a smaller LM. Central to
AutoMix is a few-shot self-verification mechanism, which estimates the
reliability of its own outputs without requiring training. Given that
verifications can be noisy, we employ a meta verifier in AutoMix to refine the
accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five
context-grounded reasoning datasets demonstrate that AutoMix surpasses
established baselines, improving the incremental benefit per cost by up to 89%.
Our code and data are available at https://github.com/automix-llm/automix.
AutoMix strategically routes queries to larger LLMs based on a noise-refined self-verification mechanism, improving the incremental benefit per cost.
AutoMix few-shot self-verification meta verifier LLMs large language models context-grounded reasoning LLAMA2-13/70B