Solved by vdw in post #12 
              
              
                You usually call init_hidden() or detach() after each batch. 
Yes, without init_hidden(), you given the last hidden state of the previous batch as first hidden state for the current batch. However, detach() ensures that the hidden state is a constant and the loss is not backpropagated to the previou…