GPT2 perplexity rolling/striding way for evaluating a document.
As I understand GPT2 uses TextDataset as a loader and it produces the example list in block sizes. So say we have a sentence "**we are in a climate crisis**" and have block size 3. So this will produce the example list as 

`ex = [["we","are","in"],["a","climate","crisis"]]`

So in such a scenario when calculating overall perplexity for the document the word "a" has no previous context and "climate" only has "a" as context. What I would want would ideally the context to be a in rolling/striding way. So  I edited the text loader to produce a list like:

`ex = [["we","are","in"],["are","in","a"],["in,"a","climate],["a","climate","crisis"]]`

Now, if I calculate perplexity for this ex list surely a lot of words will be counted multiple times as they are in the lists in multiple times. But in in a way for every instance from 2nd onwards (ex[1] above) i Would only want to consider last word for perplexity/loss calculation.  So my query is how to tackle it, should I use attention mask so that say in the case of 

`ex[1] =["are","in","a"] ` 
with mask of [0,0,1] for loss calculation (and hence perplexity) it only takes "a" into account but for getting the context of "a' will also take previous 2 words in account?

Any help on the best way to approach this problem will be much appreciated.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
