Great project! I’d also be interested in why one should use BART. Maybe T5 is also an option? We’ll have an official pretraining script merged for T5 very soon → see: [Flax] Add T5 pretraining script by patrickvonplaten · Pull Request #12355 · huggingface/transformers · GitHub 6