1We believe the LLaMa 13B model is better than Cerebras-GPT on downstream tasks because it was trained for 4× more

tokens, but were unable to get access to test the model ourselves.

©2023 Cerebras Systems Inc. All Rights Reserved.

4

Cerebras-GPT: Open Compute-Optimal Language Models

3.1 Pre-training Results
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster