The I found the reason.
If your final model output is the product of tensors all with requires_grad = False, then it is impossible to do the  back-propagation.
I had this problem as I accidentally removed the connection between my branch and model.