Head Block: We ﬁnd that we can remove the nonlinear head without ill effect. We can further
drop the decoder bias (Radford et al., 2019) and gain in memory using sparse token prediction (Liu
et al., 2019; Izsak et al., 2021). We add a ﬁnal Layer Norm to stabilize training further.

4.3 MODIFYING THE TRAINING SETUP

We study the impact of training hyper-parameters on the BERT-base architecture. The original
BERT training recipe understandably results is poor model performance in the cramming setting,
and so we revisit a number of standard choices.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY