Crammed BERT
Crammed BERT (no clipping)

MNLI
83.2/83.4
34.1/34.1
86.6/86.4
83.9/84.1

86.5/86.7
86.1/86.7

SST-2
91.9
79.9
93.7
92.2

93.8
93.2

STSB RTE QNLI QQP MRPC CoLA GLUE
80.9
45.5
85.1
78.6

56.5
0.0
60.2
44.5

90.6
50.0
92.1
89.5

87.7
68.6
88.3
87.3

89.3
77.9
91.4
87.5

86.7
17.8
90.4
84.6

59.2
47.3
77.3
53.8

Trained for 2 days on 8 A6000:

86.8
87.1

53.4
55.2

91.6
92.1

88.0
88.3

88.2
90.2

42.9
46.6

79.8
80.6

Table 6: Models trained on 16x as much compute as otherwise in this work, but with exactly the same setup
and data. With this budget, we use about half as much compute as one of the original BERT training runs. The
resulting models (which are surprisingly slightly improved by removing gradient clipping again), are equivalent
in performance, even to ROBERTA-base models trained in Liu et al. (2019), on some tasks. On other tasks,
such as CoLA and RTE, the additional compute barely improves performance.

5.2 WHAT HAPPENS WHEN TRAINING LONGER?
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY