Well, dataparallel duplicates the model in each gpu by definition. I think it’s explained in docs. Gradients are computed gpu-wise also. Think is that final gradients with which all models are updated are an average of gradients on each gpu but in the end it uses same memory than it would use if you had a single gpu.