I’m training a language model using the code here: https://github.com/pytorch/examples/blob/master/word_language_model/main.py 23
I have made some slight changes so that the model can be trained across multiple GPUs. However, the GPU memory usage is extremely imbalanced.

WX20180316-152226.png1103×191 15 KB

I can understand that one GPU is set to gather and store all outputs. I wonder if there is any way I can balance the memory usage? Or can I set one GPU for gathering outputs and the rest for training on batches?
Thanks,
Yuzhou