Hi @sgugger ,
I want to do a pre-training PEGASUS model from scratch, can you five me some suggestion?
First, can I do this approach (How to train a new language model from scratch using Transformers and Tokenizers 12) to train this model from scratch?
Secondly, how can I control the <mask_1> tokens to mask sentences (GSG objective)? And how can I specify the strategy of masking sentences like the selected one in PEGASUS paper?
Thank you very much!