I don’t think we have a really good example for seq2seq training from scratch right now, but you can take a look at the examples/seq2seq folder which implements fine-tuning, and adapt.
This is using pytorch-lightning.
@sshleifer can also chime in.