FineTuning using single sentence document
Hello

roBERTa is not using the next sentence prediction objective. I want to fine-tune the pre-trained model on an unlabelled corpus of domain-specific text (ULMFIT style intermediate pretraining).
The bottleneck is my examples are single short sentences, instead of a document with multiple sentences 

The INPUT format here says it requires a file with one sentence per line and one blank line between documents. For me, each document has a single sentence
https://github.com/huggingface/pytorch-transformers/tree/b62abe87c94f8df4d5fdc2e9202da651be9c331d/examples/lm_finetuning


The last time I raised an issue this was an expected behavior I know
Re : https://github.com/huggingface/pytorch-transformers/issues/272

How to do it now :) ? Any help is appreciated . 
Hi Tuhin, you can use `examples/run_lm_finetuning.py` now. The scripts in `examples/lm_finetuning` are deprecated (removed on master now).<|||||>Thomas i checked the wiki text 2 format and its confusing to me . Do we have to seperate documents by new lines ? My input file is a set of single sentence documents one per line . Do i need a new line after each sentence ?

My format is
sent1
sent2
sent3

Do i need to have
sent1

sent2

sent3 ?

I am currently running without a new line after each sentence and getting 09/19/2019 03:37:22 - WARNING - pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (3989110 >
 512). Running this sequence through the model will result in indexing errors


<|||||>Also while running
def mask_tokens(inputs, tokenizer, args):

- [ ] labels = inputs.clone()


   

- [ ]  # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
- [ ]     **masked_indices = torch.bernoulli(torch.full(labels.shape, args.mlm_probability)).bool()**
- [ ] 

Getting error 
AttributeError: 'Tensor' object has no attribute 'bool'
Line 110
<|||||>you need to update your pytorch version.  I believe bool() function was intrcoduced in torch 1.1.0 or 1.2.0<|||||>> I am currently running without a new line after each sentence and getting 09/19/2019 03:37:22 - WARNING - pytorch_transformers.tokenization_utils - Token indices sequence length is longer than the specified maximum sequence length for this model (3989110 >
> 512). Running this sequence through the model will result in indexing errors

I'm also getting this error. Is there more detail on the expected input format for the LM pretraining?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
