I copied and pasted that as suggested, and I am still getting nan values when it’s calculating the total norm. Line 36 of the code I copied calculates the total norm as:
total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters]), norm_type)
I did the p.grad.detach() function on a separate line and I noticed that’s where the nan values start popping up.