Thank you for your answer! This really clarifies a few things for me. Many thanks.
Looking at the code again last night, there is also one more thing that stuck out. Usually these auto encoder like things have a RNN for encoding and another one for decoding (recon). However, in this code, the decoder is just a linear layer. I wonder why that is?
self.decoder = nn.Linear(nhid, ntoken)
....
decoded = self.decoder(output)

Why should the reconstruction be a linear combination of the output? I am trying to wonder what this pytorch example is trying to achieve  with this? I also ask because I have seen similar things in RNN based anomaly detection but am confused about the role of this linear decoder layer.