@valhalla
Questions about Seq2SeqTrainer

does it know how to do multigpu sortish sampler?
does it know how to sync metrics in a multigpu setting?
is TPU faster than GPU  ?

Experiments

Would be interested to know how  finetune bart-large on xsum performs, for example, esp. on TPU .
Would be cool to make a blogpost showing some sort of optuna/ray search for summarization or translation finetuning. Would also be cool to know the best hparams.
Good hparams to check are learning rate, gradient_accumulation_steps, label_smoothing, dropout.