I have the same issue. Gradients for weight_hh_l0 in my 1 layer LSTM network are just zeros. I have checked if they are really zeros, and yes, they are simply 0s. Throughout the backward pass, they never get updated. What could be the reason for such behaviour?