D Additional Tokens-per-parameter Experiments

Here, we give more evidence that 20 tokens-per-parameter is nearly compute-optimal when pre-training
GPT-like models on the Pile dataset.

©2023 Cerebras Systems Inc. All Rights Reserved.

28

Cerebras-GPT: Open Compute-Optimal Language Models

Estimated Chinchilla Losses
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster