I’ve just spent a few days struggling with a similar issue. It seems that the variance in my data was too small. I was able to solve the issue by running the data through a BatchNorm layer prior to feeding it to my LSTM.