Large Language Models (LLMs) improve in performance by increasing both the size of the dataset and the number of parameters in the model, and the optimal quantity of either factor will vary based on the amount of compute that is at your disposal. Estimates for the optimal model and dataset sizes for a given FLOP budget using the Chinchilla scaling laws (Hoffman 2022) can be found below:
| Model Size |  FLOP Budget |  Tokens |
| --- | --- | ---|
| 4.00E+08 |  1.92E+19 |  8.00E+09 |
| 1.00E+09 |  1.21E+20 |  2.02E+10 |
| 1.00E+10 |  1.23E+22 |  2.05E+11 |
| 6.70E+10 |  5.76E+23 |  1.50E+12 |
| 1.75E+11 |  3.85E+24 |  3.70E+12 |
| 2.80E+11 |  9.90E+24 |  5.90E+12 |
| 5.20E+11 |  3.43E+25 |  1.10E+13 |
| 1.00E+12 |  1.27E+26 |  2.12E+13 |
| 1.00E+13 |  1.30E+28 |  2.16E+14 |
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence