Thanks! About the data, sequences, preprocess(padding), they are exactly the same in my original project, even thought this is a simplified one. The architecture is also the same, RNN/LSTM/GRU encoder/decoder with only 1 layer. (The simplest seq2seq net, as described in the code). When training, only the learning rate is tuned in [1e-2, 1e-3, 1e-4].
Before, in tensorflow, I tried to initialize the embedding to large value (for comparison), e.g. uniform (-1, 1) and without gradient clipping, the hidden state in tensorflow never goes to -1/1 in thousands of steps. When the PyTorch version goes to -1/1 in dozens of steps.
I personally think that both should easily overfit a very small dataset. (n_data=20/50 in the code.)