I did some more research, I found some suggestions to the problem, as @tom explains the problem of training language model with “bidirectional” LSTMs as:



 tom:

So if you take the template

Now you take a bidirectional model, feed ??? and the model outputs a probability distribution ??? and take the probability of ???.

and try to make it as precise as the I did for the unidirectional case:

after feeding c_0 … c_n, the model outputs a probability distribution p over the alphabet and perplexity is exp(-p(c_{n+1}), where we took c_{n+1} from the ground truth, you take and you take the expectation / average over your validation set.

you find that it is hard (impossible?) to do without having the target somehow showing up in the input.


To help this problem it is suggested in this blog post 33 is first to append each sentence with start and end of sentecne tags then instead of concatenating coressponding outputs of each layer, concatenate forward and backawad layers so that they are predicting on the same token.
There is also further explanations in this paper 16