@BramVanroy btw, Do you use nn.DataParallel() or nn.parallel.DistributedDataParallel?
I found that when using nn.DataParallel() torch.amp.GradScaler() not work but nn.parallel.DistributedDataParallel() work well.