Diving into the Model

GPT-3 comes in eight sizes, ranging from 125M to 175B parameters. The largest GPT-3
model is an order of magnitude larger than the previous record holder, T5-11B. The smallest
GPT-3 model is roughly the size of BERT-Base and RoBERTa-Base.

All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor.
The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension
heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-
dimension heads.

GPT-3 expanded the capacity of its GPT-2 by three orders of magnitudes without
significant modification of the model architecture — just more layers, wider layers, and
more data to train it on.

Understanding the Data

https://lambdalabs.com/blog/demystifying-gpt-3

2/11

21/08/2023, 16:10

OpenAI's GPT-3 Language Model: A Technical Overview
OpenAI's GPT-3 Language Model_ A Technical Overview