Thanks for your help and creating the issue. I have been able to solve the problem since. I really had a gradient explosion, which caused Infs (which were turned into NaNs by cudnn).
To mitigate the problem, I am now clipping the gradients in a backward of a torch.autograd.Function (register_hook would work as well). I cannot use normal gradient clipping, since it happens immediately in the first iteration.