The PyTorch official tutorials have one which mentions this training (teacher forcing) and how to train against its own distribution: http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html 1.1k. Note that even though this is seq2seq it is relevant for you as well.