Any solutions? I’m running into this issue too. It seems the model is trained but the perplexity is not calculated correctly but when I switch into unidirectional LSTM layers perplexity sounds to be a valid value.