Thank you for replying. I will try it. Can I ask something more? During the training, I found that there are same training loss regardless of number of GPUs. Code is same as above. I tried to use single, two, four GPUs, but I feel there are no differences. (I expect training more fast, but I couldn’t maintain batch size because of OOM. I think it’s because gathering loss in one GPU. As a result, training speed is decreased.) Moreover, when I use multi-gpu, many warnings are occur. How can I upgrade performance or speed with multi-GPUs?