I’m not sure why your loss would not decrease. Depending on your machine, and your initialization, LSTM can take a very long time to learn ! I remember once seeing no improvement for hours.
The hidden argument passed to LSTM is a tuple with the initial tensors (h, c). If you do not give this argument, they will be initialized to zero, which is fine.