Hi @yusukemori
The Seq2SeqTrainer examples doesn’t (yet!) support pre-training, however you could use Seq2SeqTrainer with your custom data processing logic(masking etc) as it’s a generic s2s trainer.
But you’ll need to implement masking yourself.