Perplexity for (not-stateful) Transformer - Why is it still fair to compare to RNN?
Greetings,

It is clear to me how to compute perplexity for RNNs as RNNs are a stateful model. 

In general, given a very long document, I believe we need to: 1. chunk the document into a sequence of chunks, and we compute the cross entropy loss for each of the chunks, before taking the average for all chunks and take exponential. 

Because RNNs are a stateful model, we can use what we have from the last RNN hidden state regarding to previous chunk to initialize hidden state for RNN that handles the next chunk. Because of this, I think RNN computes the "right" perplexity for a long document.

Meanwhile Transformer are not a stateful model. For this reason there is no such things as "transferring" hidden state from previous chunk to next chunk I guess. For this reason I think Transformers are not really computing the perplexity for a long document. It compute something different.

So why is it still fair to compare two models regarding to perplexity for long documents?
Plus, if I create a non-stateful RNN (e.g. as in Keras I can set stateful=False), will the perplexity the non-stateful model computes without "transferring" hidden state be the right perplexity we need? Is it comparable to the stateful RNN?

Thank you!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
