Hi, @Mendel123  thanks for your reply.
After analysis the running time of  each part, Forward has taken the largest part.





Forward
backward
Data




2 GPU(32 batchsize)
1.1 sec
0.27 sec
0.001


1 GPU(16 batchsize)
0.4 sec
0.27 sec
0.001