I use only 32 batch_size and I don’t need to size up the batch. I just want to more speed. In this situation, does DistributedDataParallel can help me to speed up? (I have 4-gpus 12,000MiB per gpu, and my code needs at least 40,000MiB)

Yes DDP can improve training speed. It will need to be configured correctly. You can benchmark your configurations to see what works best for your use case.
This paper provides helpful information on this topic https://research.fb.com/wp-content/uploads/2020/08/PyTorch-Distributed-Experiences-on-Accelerating-Data-Parallel-Training.pdf 3.