github.com
  
  
    

huggingface/transformers 22

master/examples/language-modeling

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.

  
  
    
    
  
  


I think you can start from here to finetune pegausus on the language you want, you should choose causal language modeling for pegasus if i’m not mistaken