Yes, it’s the same on CPU only. I updated the code with disabled cuDNN and CPU only.
Results are similar when using SGD. It seems a RMSProp/Adam problem but I didn’t try other Ada* algorithms.
edit: interesting fact: the issue does not appear with LSTM and GRU, on the contrary PyTorch performs even better!