I don’t know what might be causing the imbalanced memory usage, as you are pointing towards the DistributedSampler being the potential root cause while the code snippet shows the memory increase is caused after the optimizer.step() method.
Assuming the optimizer.step() method does indeed increase the memory on the default device only, it could mean that the optimizer is storing its internal running stats only on this device which wouldn’t fit the DDP use case. In case you are using nn.DataParallel, an imbalance is expected but then you wouldn’t use a DistributedSampler.