I’m sorry, I said a stupid thing. Are you reinitializing the hidden states between samples yet? If you don’t want to reinizitialize them, have you tried to detach the hidden states between batches? Usually you don’t need the entire history of the hidden states between one example and one other.