Hi @mralexis, there’s a GitHub issue that might help you: How do I pre-train the T5 model in HuggingFace library using my own text corpus? · Issue #5079 · huggingface/transformers · GitHub 340
In particular T5ForConditionalGeneration is probably what you are looking for doing pretraining: T5 — transformers 4.3.0 documentation 57