I dug into it more and it seems that, for the Pytorch version, the outputs of the RNN almost all converge to 1/-1. Seems like we are saturating the tanh activation…
It’s strange that this is occurring with Pytorch and not Tensorflow. I was also able to overfit a sample train batch on TF but not on Pytorch.
Someone had a similar problem here

  
    
    
    Hidden units saturate in a seq2seq model in PyTorch 
  
  
    I’m trying to write a very simple machine translation toy example in PyTorch. To simply the question, I turn the machine translation task into this one: 

Given a random sequence ([4, 8, 9 ...]), predict a sequence whose elements its elements plus 1 ([5, 9, 10, ...]). Id: 0, 1, 2 will be used as pad, bos, eos, respectively. 

I observed the same problem in this toy task in my machine translation task. To debug, I use a very small data size n_data = 50, and find that the model can not even overfi…
  


If you have any suggestions, please let me know! Thanks