• Risks and harms: There can be distributional bias in the Pile dataset that can manifest in various forms in the
downstream model deployment. There are other risks associated with large language models such as amplifying social
stereotypes, memorizing training data, or revealing private or secure information.

• Mitigations: Only mitigations in standard Pile dataset pre-processing were employed when pre-training Cerebras-

• Evaluation factors: Cerebras-GPT was evaluated for various bias factors using the CrowS-Pairs dataset task. Details

GPT.

Factors

are in Appendix C.4.

Implementation infrastructure

• Hardware: Andromeda AI Supercomputer: Cerebras Wafer-Scale Cluster with 16 Cerebras CS-2 systems
• Software: PyTorch, Cerebras Software Platform (CSoft) release 1.8

©2023 Cerebras Systems Inc. All Rights Reserved.

18

Cerebras-GPT: Open Compute-Optimal Language Models

Cerebras-GPT Open-Source References
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster