From transformers 3.x, I have been using examples/seq2seq/finetune.py (huggingface-transformers/finetune.py at v3.5.1 · microsoft/huggingface-transformers · GitHub 4) to finetune pegasus model and it’s been working fine.
After upgrading to transformers 4.x, that script has been moved to legacy and/so I’m thinking of using the examples/seq2seq/run_summarization.py (transformers/run_summarization.py at v4.4.2 · huggingface/transformers · GitHub 1) for the same training.
I moved pieces around to make the old finetune.py work under transformers 4.x as well to have fair comparison. Other than the dataset format difference between the two, I mainly noticed the huge difference in training speed between the two.
For the same training dataset (~6 million data points):


finetune.py: with 4 v100 GPU, taking ~6.5h/epoch.

run_summarization.py with 4 v100 GPU, taking ~15h/epoch.

Upon reading the code, finetune.py from transformers 3.x uses pytorch-lightning but run_summarization.py I believe just uses pytorch. Is this mainly causing the difference in speed? Or are there any signification implementation difference between the two?
I can provide more details if needed.
Thanks!