Hey @khalil
How large is your model?

My batch_size is 1 and datasets include 4000 samples,every sample is about 16MB.

Does this actually mean the memory is consumed by the data loader? Can you check the memory consumption when using the same model and same data loader without DDP?