ng.

2

Preprint

Accelerator

Time Limit

Total exaFLOP

Group

(Devlin et al., 2019)

(Dettmers, 2018)
(Narasimhan, 2019)
(Raffel et al., 2020)
(Iandola et al., 2020)
(Narang et al., 2021)

(Tay et al., 2021)
(Izsak et al., 2021)
(Liu et al., 2019)

Our Setup 1
Our Setup 2
Our Setup 3

Target
BERT
BERT

BERT-large

T5-base

squeezeBERT
T5 variations
T5-small-L16
BERT variation
roBERTa-base

BERT variation
BERT variation
BERT variation

(Chowdhery et al., 2022)

PaLM

16 TPU
8 V100

1472 V100
16 TPUv3

8 Titan RTX

16 TPUv3
16 TPUv3

8 V100

1024 V100
6144 TPUv4
1 rtx2080ti
1 rtxa4000
1 rtxa6000

4 days
11 days
47 min
1 day
4 days

1.75 days
11.2 hours

1 day

1.25 day
50 days
1 day
1 day
1 day

680
950
519
170
361
298
82
86

5
8
13

13 824
7 299 072

Table 1: Maximal Throughput available for select training runs of large language models. FLOP Counts for
BERT reproductions and related models. Large-scale LMs included only for reference.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY