Slow training time on BERT pretraining on multiple gpu compare to single gpu
# ❓ Questions & Help

Hello, I'm pretraining BERT on two RTX 2080Ti with 15 batch size on each gpu. They update every 40 steps.

The training time to 28000 steps of 2 2080Ti is around 9 days and it only takes 1 2080Ti 10 days to achieve the same training time. I was wondering whether is training time is accurate or I can do something to improve it. 

First problem I can think of is although I expand to batch size to the most one GPU memory can hold. The gpu usage is only around 50% all the time. I'm not sure how to improve the overall gpu usage, is it possible that 50% usage is normal? Please give me some advice.

Sincerely
I solve this problem with distributed training.

The DataParallel is really slow with a lot of defects, just don't use it.

<|||||>Hi @ntubertchen, how did you make distributed training to work with the `run_language_modeling.py` script? Or did you do everything from scratch?<|||||>run_language_modeling.py has written the distributed part, just use the command.