Thanks for the help. I tweaked the model a little bit so the encoder and decoder shares the same embedding since it’s not supposed to be trainable. Now I am able to do 20 bs on one single GPU. But when I do it with DataParallel, I am unable to increase my batch size. I tried 24, which already gives me an error. By observing the memory usage, the core gpu that the results are collected would blow up…