You get 5 hidden states since you have 2 sequences of length 5: [1,2,3,4,5] and [4,5,6,7,8]. 2 is here your batch size. You do not have a sequence of 2 words, but a minibatch of 2 sequences (representing words), each with 5 characters. The LSTM will loop over the 5 characters yielding 5 hidden states for each sequence.
Please note, that there’s no semantic when you say that a minibatch is a sentence. Sure, all the words in the minibatch may come from a single sentence, but for the LSTM it’s just bag of words with no connection.
LSTM can handle variable sequence lengths just fine as long all sequences in the batch have the same length. This is why padding might be needed. For example, you cannot have [1,2,3,4,5] and [4,5,6] in the same batch since they have different lenghts, but you could pad it to, e.g., [4,5,6,0,0]. Also note that if your minibatches have sequences with different lentghs, you need to re-initialize your hidden state w.r.t. the the current sequence length of the current batch (for an example, see the init_hidden method here 18).