The Cerebras-GPT models exhibit less bias on average than any of the larger model baselines. However,
Cerebras-GPT 13B does show bias greater than GPT-3, OPT, or LLaMa on six of the nine bias categories,
indicating that compute-eﬃcient pre-training is not immune to large bias.
When observing bias levels across Cerebras-GPT or Pythia models, we see that biases tend to grow with
model size. The Cerebras-GPT models tend to show a larger range of bias values over the growing model
sizes (e.g., gender), while Pythia models sometimes show similar bias across model sizes (e.g., disability).
This suggests that models trained on a ﬁxed dataset size may be likely to extract similar levels of bias
regardless of model size. On the other hand, more compute-eﬃcient training (smaller datasets for smaller
models) might mitigate some bias issues compared to models trained on more data.
Finally, we note that when comparing Cerebras-GPT and Pythia models trained with similar compute
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster