Fix bug in run_squad.py
I fired an issue in Google's repo, https://github.com/google-research/bert/issues/540#issue-428344784

After testing, I found it is indeed a bug.
We should not put these chunks with `start_position==0` and `end_position==0` into training set.

Thanks for code review.
Hi @MottoX, sorry for the delay on reviewing this.
It seems to make sense to me.
What kind of testing are you referring to? Could you share a bit more about them?<|||||>Hi, @thomwolf 
I was trying to train a BERT-based model on NewsQA, a document-level QA dataset, using `run_squad.py`. I used 384 for max_seq_length and 128 for doc_stride. The model got an extremely bad result. I found there were numerous chunks with start_position and end_position as zeros.
Then I checked the implementation of `run_squad.py` as well as `BertForQuestionAnswering`, which is basically BERT + one fully connected layer. I suspected that such out-of-span chunks are harmful for model to converge. So I modified the script to drop all these out-of-span chunks during training and got an acceptable result. I am sorry that I do not remember the exact numbers before and after doing this.

I think the reason why the problem is not identified before is that context length (token-level) in SQuAD is not that long, which means the model can still easily converge anyway, with or without out-of-span chunks during training as they are really rare. However, when we have numerous out-of-span chunks in the training set, there will be a problem for model to converge.<|||||>I looked at the code again, and found it is quite confusing, since chunks with `start_position==0` and `end_position==0` are also used for the cases where we have examples that do not have answer (for version_2_with_negative). So the current fix seems not compatible with this. Now I am not exactly sure whether we need to train the model with those out-of-span chunks. However, from my last experience, there should be a problem when we have too many such out-of-span chunks in the training set and the model is biased to predicting the result on the first token (CLS), despite the fact that `write_predictions` filters out those invalid predictions.