Fine-tuning Bert
I want to fine-tune Bert's LM for a specific corpora. I converted the test into the format specified in the documentation and ran the fine-tuning codes given. I'm getting the following error:
File "simple_lm_finetuning.py", line 156, in random_sent
    assert len(t2) > 0
AssertionError
I'm getting similar error in pregenerate_.... script. What could be the reason for the same? Is it due to some possible OOV words? My corpus does contain some emoticons.
Thanks in advance
I was facing the same issue when finetuning using `finetune_on_pregenerated.py`. The problem was in the fact that I have some empty sentences in my dataset. Also there are some special characters, like `\t` (tabulation) which can make a mess and should be cleared. 
I preprocess the text like this:
```
for_train = for_train.dropna(subset=['text'])

import re
with open('for_pretraining_full.txt', "w", encoding='utf-8') as writer:
    for doc in for_train['text'].tolist():
        doc = doc.replace(u'\xa0', u' ').replace(u'\u200b', u' ').replace(u'\u206f', u' ').replace(u'\u206e', u' ').replace(u'\u206b', u' ').replace(u'\u206c', u' ').replace(u'\u2063', u' ').replace(u'\u200d', u' ').strip() # replace some special unicode chars
        doc = re.sub('\t+', '', doc) # replace tabs
        doc = doc.replace('. ', '\n')
        doc = re.sub('\n+( )*(\n+)*', '\n', doc) # replace several consecutive new lines by a single one
        doc = doc.strip()
        if (doc != ''):
            writer.write(doc)
            writer.write('\n\n')
```

Also you can try to find problem sentances, debuging your `simple_lm_finetuning.py`.<|||||>I ran into an issue on this where some sneaky \n were still sneaking through even with the above code. I remedied this by just doing 

`if "\n" in doc:

    doc = re.sub('\n', ' ', doc)`<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
