Do you re-initialize your hidden sate after each batch? See this Seq2Seq PyTorch tutoria 25l, specifically the use if the initHidden() methods of the encoder and decoder.
Without re-initalizing (or the usage of detach() at the right spot), the backpropagation path of your RNN continuously grows, definitely leading to memory issues.