Using doc chunks without answer token during training ( BertForQuestionAnswering )
Hi!

When creating features from the squad examples, you use sliding window approach to generate doc chunks, when the input data is too long. There you state, **_if the document chunk does not contain an annotation, you throw it out, since there is nothing to predict._** Hence you set the start_position=0 and end_position=0. Then you re-set it to cls_index in training mode. When you do **CLS token at the beginning, then your cls_index = 0**. Finishing up, you add this item to the InputFeatures. 
https://github.com/huggingface/transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/utils_squad.py#L332-L351

During training - forward step, you calculate loss on these items:
Let's say the max seq length is 512, then the ignored_index = 512, meaning you only ignore those start/end positions which are >= 512. With cls_index = 0 we have start_position=0 and end_position=0. So we end up getting a loss calculated using the start_position with start_logits and end_position with end_logits.
https://github.com/huggingface/transformers/blob/a8c3f9aa760ed7b516ee00f602e8efc0e5d80285/src/transformers/models/bert/modeling_bert.py#L1651-L1665

Then **you return with the calculated loss and in run_squad.py you add this new loss to the training loss**:
https://github.com/huggingface/transformers/blob/a8c3f9aa760ed7b516ee00f602e8efc0e5d80285/examples/question-answering/run_squad.py#L219

**So actually you do not throw these chunks out, but use them for training.** 

**Solution maybe:**
Why not set the start_position = end_position = **max_seq_length** instead of cls_index in the utils_squad.py, making sure they will be ignored during training and the loss calculated with them will be 0 ??

Hope you will understand my point, let me know what you think!

**Update:**
I got this idea from another repo using bits and pieces from your implementation (they only used labels with answer, so squad-v1 like dataset) :
In case of too long sequences the sliding window approach creates multiple doc chunks, some of those are without an annotated answer. The model learns to predict start and end positions as 0  (if CLS is on 0. pos) when the answer is NOT present in the document chunk. This will help reduce false predictions on test data where documents are too long and split into multiple chunks.

**So either you actually want to use those doc chunks with no annotation, but your comments are misleading, 
OR you don't want to use those chunks, hence the comment, but you failed to implement it.**

Hope it helps!

@sgugger you might be interested in that issue given that you're refactoring the squad example!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.