C.3 Diﬀerences Between Cerebras-GPT and Other Models

The Cerebras-GPT 13B model improves over other publicly-available models of comparable size. This is a
surprising result given that the creators of these other models modiﬁed the original GPT-2/3 architecture
intending to improve convergence and training eﬃciency. There are many confounders that could contribute
to Cerebras-GPT’s advantages, but we brieﬂy list known diﬀerences here to give an idea of the space of
possible opportunities for future study.

• GPT-J, GPT-NeoX, and Pythia models use rotary positional embeddings, which show modest
loss/accuracy improvements and ability to extend to longer sequence lengths (Su et al., 2022).
Cerebras-GPT uses standard trainable positional embeddings.
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster