Even I am facing the same problem in LSTM. My gradients are of the order of 10^-9. Therefore my model is not learning anything and predicting same class in each iteration. Can someone please tell a solution?