An unrelated issue…
optimizer.zero_grad should come before loss.backward or after optimizer.step. If you put it after .backward and before .step then you delete the gradients needed for the update before applying the update.
LogSoftmax outputs values between -inf and 0. I suspect that with each step your hidden state is getting closer and closer to -inf.
You might have better luck if you use a bounded activation for the hidden states, such as nn.Tanh.