I think you should initialize the cell state and hidden state to a tensor of zeros, instead of randn.
These are not like weights, if you initialize these randomly then the forget gate in the LSTM cell will try to remember stuff that it’s not relevant since this is the first timestep in your sequence.
The ‘forget gate’ is a sigmoid layer that decides what is relevant in each timestep. Outputting a value between 0 and 1, with 0 meaning ‘completely forget this’ and 1 meaning ‘this is really important so lets remember it’.
So if you use randn for the hidden state initialization. These random numbers will be fed into the forget gate and the cell state, adding noise to the LSTM cell, maybe this is what is preventing convergence in your model. But when you initialize them to 0 then the forget gate just ‘forgets’ them and the cell does not try to remember random stuff.