@gcramer23  Thanks for your reply. There was no error but, I just wondered that my code is working fine as I intended.
I use only 32 batch_size and I don’t need to size up the batch. I just want to more speed. In this situation, does  DistributedDataParallel can help me to speed up? (I have 4-gpus 12,000MiB per gpu, and my code needs at least 40,000MiB)