mrshenli:

It depends on when do you wants to inspect the memory usage. It prints current cached memory, allocated memory, etc. I would try to print it every few iterations.


I’ll get on that tomorrow and post the results.



 mrshenli:

Are you using PyTorch v1.6+? I saw the DDP + AMP example is only available in v1.6+ docs:
https://pytorch.org/docs/1.5.0/notes/amp_examples.html
https://pytorch.org/docs/master/notes/amp_examples.html
cc the author of torch.cuda.amp @mcarilli


yes I tested both 1.5 and 1.7, and saw speed up when using 1.7 with AMP in both single GPU mode and DataParallel mode, but not in DistributedDataParallel.
Should I @ him aswell or is yours sufficient?