What do you mean by “pretrained hidden state”?
The hidden state of an RNN (vanilla, LSTM, GRU or any other kind that you may like) is not supposed to “live forever”. In principle, it should be reset to an empty state (all zeros) after a whole sequence has been processed.