[Bug?] question answering - end position of each input is weird
I run "python run_qa.py" in transformers/examples/pytorch/question-answering.
In prepare_train_features function, I think "end position" is lower than expected postiion.

I tested first example in examples(squad) in "prepare_train_features" function
For example, answer text = 'Saint Bernadette Soubirous'
print(tokenizer(answer_text))
=> return: {'input_ids': [101, 3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

print(input_ids[tokenized_examples['start_positions'][0]:tokenized_examples['end_positions'][0]])
=> return: [3002, 16595, 9648, 4674, 2061, 12083, 9711]

=> Thus I think last token 2271 is dropped. 

For other input sentences, I think last token is dropped
Isn't it bug??
I believe @sgugger worked on that script<|||||>Hi there, noticed you closed this so may have come to the same conclusion, but the "end_positions" will give you the position of the last token in the answer. So you should add a +1 in your slice to include that token at "end_positions".<|||||>Thank you for your reply.
I recognized my mistakes, thus I closed the issue myself.
Thank you for checking one more time.
Next time, I'll post the issue deliberately!

2021년 7월 16일 (금) 오전 2:13, Sylvain Gugger ***@***.***>님이 작성:

> Hi there, noticed you closed this so may have come to the same conclusion,
> but the "end_positions" will give you the position of the last token in the
> answer. So you should add a +1 in your slice to include that token at
> "end_positions".
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12708#issuecomment-880872726>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOQH5YFSBA4AWDZPF77K3TLTX4JMJANCNFSM5AL5BOQQ>
> .
>
