Answers.Body: <p>The minimum number of test items is a function of the statistical power you want to reach so that you can reliably detect differences of a given magnitude. A small number of test items (a small sample size) can be acceptable for some purposes but be too small for other purposes.</p>
  <p>In some fields of research, there are traditionally accepted test set sizes, for example 3000 items. In such fields, you have a good chance of getting your paper published (and pass your PhD exam) with a test set of such a size even if some publications have shown that the size is too small for the task.</p>
  <p>If your task is to pick the best model from a large set of candidate models with a limited inference budget, you could first test all models on a small subset of the test set, select the best models (the optimal number of models to select depends on many factors) according to this incomplete test, then test the selected models with more data and repeat the process. Whether the final round uses the full test set and how many models should be in the final is also hard to say. Ideally, you'd test some candidate selection procedures using small, inexpensive LMs and then scale up to your LLMs. An ad-hoc approach like going from 1000 models to 100 models using a test set of 1000 items, then 100 to 10 using 7000 items and finally 10 to 1 using all 40k items can be acceptable in many contexts.</p>
  <p>Suggested reading:</p>
  <ul>
  <li>Bowman and Dahl (2021) <a href="https://aclanthology.org/2021.naacl-main.385/" rel="nofollow noreferrer">What Will it Take to Fix Benchmarking in Natural Language Understanding?</a></li>
  <li>Card et al. (2020) <a href="https://aclanthology.org/2020.emnlp-main.745/" rel="nofollow noreferrer">With Little Power Comes Great Responsibility</a></li>
  <li>Kiela et al. (2021) <a href="https://aclanthology.org/2021.naacl-main.324/" rel="nofollow noreferrer">Dynabench: Rethinking Benchmarking in NLP</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T11:54:50.753Z
Answers.LastActivityDate: 2024-03-06T12:32:43.113Z
Answers.LastEditDate: 2024-03-06T12:32:43.113Z
Answers.UserDisplayName: Joachim Wagner
Body: <p>I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. My current plan is to implement an early stopping threshold.</p>
  <p><strong>Is there an &quot;accepted&quot; or theoretical minimum number of samples that are required to benchmark an LLM?</strong></p>
  
Tags: <natural-language-processing><large-language-models><benchmarks>
Title: Minimum Number of Samples for LLM Benchmark?
UserDisplayName: Jonathan Emmons
ThreadText: # Minimum Number of Samples for LLM Benchmark?
  
  (Asked by: username_0 on 2024-03-04)
  
  I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. My current plan is to implement an early stopping threshold.
  
  **Is there an "accepted" or theoretical minimum number of samples that are required to benchmark an LLM?**
  
  **Tags:** natural-language-processing, large-language-models, benchmarks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-06. Score: 1
  
  The minimum number of test items is a function of the statistical power you want to reach so that you can reliably detect differences of a given magnitude. A small number of test items (a small sample size) can be acceptable for some purposes but be too small for other purposes.
  
  In some fields of research, there are traditionally accepted test set sizes, for example 3000 items. In such fields, you have a good chance of getting your paper published (and pass your PhD exam) with a test set of such a size even if some publications have shown that the size is too small for the task.
  
  If your task is to pick the best model from a large set of candidate models with a limited inference budget, you could first test all models on a small subset of the test set, select the best models (the optimal number of models to select depends on many factors) according to this incomplete test, then test the selected models with more data and repeat the process. Whether the final round uses the full test set and how many models should be in the final is also hard to say. Ideally, you'd test some candidate selection procedures using small, inexpensive LMs and then scale up to your LLMs. An ad-hoc approach like going from 1000 models to 100 models using a test set of 1000 items, then 100 to 10 using 7000 items and finally 10 to 1 using all 40k items can be acceptable in many contexts.
  
  Suggested reading:
  
  * Bowman and Dahl (2021) [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://aclanthology.org/2021.naacl-main.385/)
  * Card et al. (2020) [With Little Power Comes Great Responsibility](https://aclanthology.org/2020.emnlp-main.745/)
  * Kiela et al. (2021) [Dynabench: Rethinking Benchmarking in NLP](https://aclanthology.org/2021.naacl-main.324/)
  
  
  ---
  
  