Hi,
The problem is that the hidden layers in your model are shared from one invocation to the next. And so they are all linked.
In particular, because the LSTM module runs the whole forward, you do not need to save the final hidden states:
lstm_out, _ = self.lstm(x.view(-1, 1, 3),self.hidden)