@jpeg729:  you have to run it with more epochs. It is normal that with a larger batch (the entire sequence in this case) it converges slower. But it converges. Try with 500 epochs.
EDIT: I found the bug. It was just the reshape ordering that must be set to Fortran-like in PyTorch (because we reshape to TxBx* instead of BxTx*.