It might be. You should inspect the gradients. Use retain_grad() to store the grad attribute on intermediate variables and look at the value.
https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.retain_grad 23