Did you insert torch.no_grad by accident or cast through integer tensors somewhere?
One simple approach to debugging this kind of thing is to print various intermediates and check that they have the “grad_fn=…” information (or just print t.grad_fn or t.requires_grad for them ). The operation that takes things requiring gradients and outputting things that don’t will be the one that trips you up.
Best regards
Thomas