Well, I had a look at the code. While I’m not familiar with this setup for a Language Model (LM) either, a look at the training data made a bit clearer to me. Still, everything that follows are not much more than educated guesses.
Most fundamentally, an LM aims to predicts the next word given an input sequence of words. Most LM architectures reflect this “directly” in the training date. For example, given a long document like “A B C D E F G H I J K L M N O P …”, the training data (input => output) is generates as (assuming an input sequence of 5)
A B C D E => F
B C D E F => G
C D E F G => H
D E F G H => I
...

So yeah, in this case you would simple pump input through the RNN and use the hidden state as input for the final linear layers, e.g.:
output, hidden = self.rnn(emb, hidden)
# optional dropout or what not
decoded = self.decoder(hidden)

That essentially simply an RNN-based classifier where the classes are all the word in the vocabulary.
The architecture here works differently. If I understand in example/main.py currectly, the training data is generated differently:
A B C D E => B C D E F
...

(the structure of subsequent training examples) depend on the batch size; check the methods batchify() and get_batch(). Anyway, the output sequence is the input sequence shifted by one word to the left.
No they treat the whole training as a sequence labeling task, which is kind of a prediction for/after each time time step and not just the last. As @ptrblck said, output[i] is the hidden state of step i. I always use the figure below to check if I use the right return values of an LSTM/GRU:

lstm-output-vs-hidden640×548 20.9 KB

Anyway, that’s how understand the proposed architecture. I assume it yields better results compared to more traditional architectures. I hope that helps…well, at least gives some food for thoughts.