I have the same question. Making an LSTM stateful in Keras requires just setting the stateful = True parameter when creating it. However, in Pytorch since the hidden states have to be managed manually, I’m unclear on what to do as I face errors when doing it manually.
My procedure is roughly:

Reset hidden state at the beginning of each epoch, and run steps 2-4 for each minibatch:
Run the LSTM forward pass for a minibatch, with lstm_out, self.hidden = self.lstm(input, self.hidden)

Call backward(retain_graph=True)

Call optimizer.step() and zero out gradients

However, when I call optimizer.step() then it throws an inplace operation error. I’m inclined to believe it’s the optimizer that’s causing the inplace error as running steps 1-3 without the optimizer results in no problem at all.
My questions, thus, are:

Is there something that I’m missing in making a stateful LSTM here?
Which of the following is equivalent to calling the LSTM with the entire sequence (since it’s too long to fit in memory):

Running the forward pass (step 2) over the entire sequence but calling loss.backward() for only the last loss
Calling loss.backward() after the forward pass (i.e. calling steps 2 and 3 for each batch) but calling optimizer.step() (step 4) at the end of each epoch?



I think the second option is equivalent to running it over the entire batch, but I’m not sure. Can anyone help clarify my thoughts on the matter?