Thanks, it makes sense. So at each stage LSTM encodes implicitly the sequence so far, e.g. A, AB, ABC, etc. So If I had a batch of 2: (D|A, B,C) and (E|B, C,D), I would have these sequences for a fourgram:
Batch1:
A->A
B->AB
C->ABC
batch 2:
B->B
C->BC
D->BCD
So If I were to build a language model, I’d need a total of 2x4xvocabulary size labels: (A, B, C, D) and (B, C, D, E) rather than (D) and (E).