Pregenerating data requires multiple documents
The script for pregenerating language modelling data assumes that the training corpus consists of multiple documents (i.e. a single training corpus file where empty lines separate documents). If the training corpus is made up of only one long text, the pregen script produces empty output.

I have a small fix for this locally, which enables pregenerating from a single document. I could create a PR if the empty output is not seen as expected behaviour.

(I can also mention that the **Installation** section of the README states that the repo has been tested with python 3.5+. However, the pregen script uses f-strings, which are a python 3.6 feature.)
Hi, I wrote that script! That's a tricky issue, though - what behaviour do you expect when your data is one long text? 

In the original BERT repo, they used the document breaks to control sampling for the NextSentence task - 'random' next sentences were selected from a different document. I'm not sure there's a "general" solution to the case when all the data is a single document, though - people whose data is all one contiguous document will probably have different expectations for how they want that data to be handled (e.g. is it okay to select a 'random' sentence 10 sentences away, or should it be much further? Are 'nearby' sentences too similar?)

I suppose we could add code to support that with a controllable parameter to stop it sampling random sentences that are too close to the actual sentence, but it might be easier just to throw an error if the data is only one single document and request that the user break it into 'documents' based on whatever structure is in the data, like chapter or page breaks, or in the worst case just add a break every 1,000 lines or something.

You're totally right about the f-strings, though. I'll look into updating the README!<|||||>I'm assuming that the script should just grab any random sentence in the same (only) document. I could see how that particular behavior might not be desired however.

The error-throwing idea is good. I ran the script on my mammoth document, only to see empty output after 9 hours (my own fault obviously, but an exception might help similarly sloppy people).<|||||>Added PR #478 to address this. Unfortunately, it probably still won't resolve your 9 hour problem - the script will only realize there was only one document in the corpus after it's finished reading and tokenizing the whole thing! Still, at least people will get an error message properly explaining the issue and suggesting fixes now.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
