You could rerun the code with torch.autograd.set_detect_anomaly(True) as suggested in the error message to get a hopefully helpful stack trace.
I cannot find the offending operation by skimming through the code.