Solved by tetratrio in post #4 
              
              
                No problem, hopefully we can find the problem! 
You just need to make sure you detach before using the hidden state again. It does not matter if you do it when you store the hidden state in the forward pass or when you retrieve the previous hidden state and feed it together with the new input to the…