I would recommend to check the Python code, especially the forward pass, which would create the grad_fns for all tensors, where the gradient calculation is not disabled.
Based on your description, did you use torch.set_grad_enabled in any part of the code or a with torch.no_grad() wrapper accidentally for some epochs?