siheming:

when should I call it ? as soon as the memory is filled or after training? or does it not matter?


It depends on when do you wants to inspect the memory usage. It prints current cached memory, allocated memory, etc. I would try to print it every few iterations.

Mixed precision has no effect on training speed (even though I observed on the GPUs that the required ram was decreased compared to not using it, and similar to the ram required for the mixed precision during DataParallel).

Are you using PyTorch v1.6+? I saw the DDP + AMP example is only available in v1.6+ docs:
https://pytorch.org/docs/1.5.0/notes/amp_examples.html
https://pytorch.org/docs/master/notes/amp_examples.html 1
cc the author of torch.cuda.amp @mcarilli