Solved by vdw in post #2 
              
              
                Do you re-initialize your hidden sate after each batch? See this Seq2Seq PyTorch tutorial, specifically the use if the initHidden() methods of the encoder and decoder. 
Without re-initalizing (or the usage of detach() at the right spot), the backpropagation path of your RNN continuously grows, defin…