mrshenli:

Can you print memory summary using  torch.cuda.memory_summary ?


when should I call it ? as soon as the memory is filled or after training? or does it not matter?
Also I wanted to come back to this question because I have not seen a satisfying answer yet or understood why this might be the case:



 siheming:

Mixed precision has no effect on training speed (even though I observed on the GPUs that the required ram was decreased compared to not using it, and similar to the ram required for the mixed precision during DataParallel).