Yes, it uses DistributedSortishSampler for multi-gpu, it’s handled here 10


Yes, metrics are handled by Trainer and  logged in the global  main process. pinging @sgugger for confirmation. TPU  training is multi-device, so for example you see TPU logs here for distilbart-cnn-12-3.



(Note: as generate   crashes/hangs on TPU we don’t calculate BLEU, ROUGE metrics on TPU)


Yes,  I’ve observed 3-4x speed boost on TPU v3-8 compared to single V100


distilbart-cnn-12-3, 1 epoch

on TPU V3-8: 25.5 min
on V100: ~102 min


marin-enro-6-3 , 1 epoch

on TPU v3-8: ~5min
on V100 : 21 min



All experiments are logged in this wandb project 6 if you want to compare yourself .

Coming soon