Answers.Body: <p>The most straightforward solution would be to define &quot;difficulty&quot; as the accuracy of a model on a dataset. If you evaluate all models on a dataset, then you can create subsets of data where e.g., all models can answer accurately, only the k-largest models can answer accurately, only the largest model can answer accurately, etc.</p>
  <p>This then gives you a supervised classification task where the goal is to map inputs to the difficulty of the sample. You could train a model to do this or try to manually come up with some heuristics.</p>
  <p>Another option is to look at current research on LLM calibration/uncertainty. There's a lot of work on getting &quot;confidence&quot; estimates from LLMs (and deep learning models, more broadly). You can then start with the smallest model, then move onto larger models only if the current prediction is deemed unconfident. This kind of stuff is definitely in the realm of on-going research rather than a technique that'll be reliable out of the box, but it could be something to look into.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T05:04:41.253Z
Answers.Text: Thanks @alexander-wan, your suggestions sound reasonable.
Answers.UserDisplayName: Xiao-Feng Li
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-22T16:43:56.190Z
Answers.LastActivityDate: 2024-06-22T16:43:56.190Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I have multiple instances of a same LLM architecture in different sizes from 3B through 72B. I want to build a system that simple questions will be sent to smaller models, and hard questions to bigger models.</p>
  <p>How to design such a system? Thanks.</p>
  <p>It is ok if it requires to send the question to the 3B model at first sometimes to accumulate experiences.</p>
  
Tags: <ai-design>
Title: How to choose the right size LLM for inference to balance latency and accuracy, among several LLMs from 3B through 140B?
UserDisplayName: Xiao-Feng Li
ThreadText: # How to choose the right size LLM for inference to balance latency and accuracy, among several LLMs from 3B through 140B?
  
  (Asked by: username_0 on 2024-06-15)
  
  I have multiple instances of a same LLM architecture in different sizes from 3B through 72B. I want to build a system that simple questions will be sent to smaller models, and hard questions to bigger models.
  
  How to design such a system? Thanks.
  
  It is ok if it requires to send the question to the 3B model at first sometimes to accumulate experiences.
  
  **Tags:** ai-design
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-22. Score: 1 | Accepted answer
  
  The most straightforward solution would be to define "difficulty" as the accuracy of a model on a dataset. If you evaluate all models on a dataset, then you can create subsets of data where e.g., all models can answer accurately, only the k-largest models can answer accurately, only the largest model can answer accurately, etc.
  
  This then gives you a supervised classification task where the goal is to map inputs to the difficulty of the sample. You could train a model to do this or try to manually come up with some heuristics.
  
  Another option is to look at current research on LLM calibration/uncertainty. There's a lot of work on getting "confidence" estimates from LLMs (and deep learning models, more broadly). You can then start with the smallest model, then move onto larger models only if the current prediction is deemed unconfident. This kind of stuff is definitely in the realm of on-going research rather than a technique that'll be reliable out of the box, but it could be something to look into.
  
  > **Comments:**
  > **username_0**: Thanks @username_1-wan, your suggestions sound reasonable.  
  
  ---
  
  