Hi, did you solve this issue yet? It seems like I ran into same issue.
I trained my model with 8 GPUs and the first one always gave me OOM error. So I moved my model to the last 7 GPUs and left the first GPU only for calculating the loss and call loss.backward. But this still costed abnormally large memory in the first GPU.
I have a embedding layer (95000 x 1024) and a decoder layer shares the same parameter with the embedding layer. When batch size is 16, the output size is 16 x 255 x 9500, it consumed 9000 MB memory in the first GPU.