Objective: We train with only masked language modeling on fully packed blocks of tokens with a
masking rate of 15% and the original setup of Devlin et al. (2019) where 10% of all masks are ﬁlled
with random words and 10% unchanged. We see no improvement from masking at larger rates,
e.g. at 40% as proposed in (Wettig et al., 2022), see Appendix. We see no difference enabling or
disabling the mentioned 20% rule. We evaluate other functions for the masked-language objective,
such as mean-squared error (Hui & Belkin, 2021) or L1 loss, but ﬁnd no beneﬁts.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY