Indeed! Typically model performance scales with parameter count and number of tokens as a power law. A power law is a relationship in which a relative change in one quantity gives rise to a proportional relative change in the other quantity, independent of the initial size of those quantities. Here are some seminal papers that investigate how scaling dataset and model size affects the performance of large language models.

  - Scaling Laws for Neural Language Models (Kaplan 2020)
  - Training Compute-Optimal Large Language Models (Hoffman 2022)
  -  Scaling Language Models: Methods, Analysis & Insights from Training Gopher (Rae 2021)
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence