 Pre-Training  The checkpoint was pretrained on the [Colossal, Cleaned version of Common Crawl (C4)](https://huggingface.co/datasets/c4) for 524288 steps using  the span-based masked language modeling (MLM) objective.  