tom:

As a practical example, when I last looked fast.ai trained separate forward and backward LMs and then evaluated the perplexity on either.


Thanks for your help. I just don’t understand how do we can train separate forward and backward model and evaluate perplexity on both. I have seen some posts and comments in forums that also mentioned about separate training.
When taking bidirectional model we feed the network same input as the unidirectional model, so the input of LSTM is shared between the forward and backward layer then the output of these layers could be summed, average or concatenated to be passed through next layers or final(dense) layer of model that produces output probabilities. We can use these probabilities and the target vector to calculate perplexity. I wanted to asked whether my interpretation is correct?
Thanks.
P.S: Sorry for not referring the link I forgot to do it. I follow the right habit next times.