Pretraining QA corpora from scratch with sentence pairs
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I would like to pretrain a non-factoid QA corpora (question and answer passages) from scratch using BERT. I have looked at both:

https://huggingface.co/blog/how-to-train and 
https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b#file-smallberta_pretraining-ipynb

I would like to confirm that what I am doing is correct: 

1. I concatenated the question and answers with a [SEP] token, so each line in my input data file looks like:
question [SEP] answer
2. I am running the script with --line_by_line

I am uncertain if this is correct because how would the script know which is sentence A and which is sentence B, or is this not necessary? 

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
