You could check the gradients in both models separately before and after the backward call via:
for param in SRmodel.parameters():
    print(param.grad)
for param in classifier.parameters():
    print(param.grad)

These should be None before the first backward call, zeros after zeroing out the gradients via optimizer.zero_grad(), and contain valid values after the backward call.