Can you check how long the individual batches require? Particularly if time time increases from batch to batch.
Most LSTM/GRU examples I see – and what I usually do as well – is to manually reset the hidden state for each batch. For example, have a look at the PyTorch Seq2Seq Tutorial 13; search for the initHidden() method and when it’s called. However, you call x = self.lstm(x) without explicitly giving the hidden/cell state as input. I assume this implies that the hidden/cell state is automatically re-initialized. The thing is, if the hidden/cell state is not re-initialized, your computational graph for backgropagation grows and grows, which certainly causes performance issues.