Thanks for the reply. I did find this, but it leaves a lot of questions unanswered:

Should the text be pretokenized? (from the example they showed it seems yes, but GPT-2 has been trained to treat spaces as tokens so this may not be ideal)
They do not use a sliding window
BOS/EOS are only inserted based on the original data. So sometimes EOS after a title sometimes after a paragraph. Wouldn’t it make more sense to have these sentence-segmented, i.e. EOS after every meaningful sentence