Terrible, converges to very high loss (start: 4, converges:2). I have also tried to force the model to over-fit by increasing the layers and hidden units. Yet without teacher forcing the model doesn’t improve. I’m at a bit of a loss, the only other thing I can consider is:
Change internals:
Deviate from the tutorial, instead of feeding only the final GRU/LSTM h_state to the decoder. I could try feeding in each encoder layer h_state to the corresponding layers in the decoder.
Change model paradigm:
Alternatively, I would like to experiment  with feeding the entire input-target sequences into the model and then retrospectively make another char-level decoder and transfer the weights of this decoder from the full model.