I checked if the offsets correctly map back to the original tokens and found out that  the start_position and end_position are sometimes being wrong by 1 or 2 tokens (around 6000 examples out of 14K were wrong)
This might be caused by the offset mappings being wrong by few characters.
I then selected a failed example and checked if the offsets map back to the original tokens nad everything looked correct.
For now, I solved the issue by applying some preprocessing and cleaning to the data which solved the issue, and reduced error count down to ~600. I tried training and got good results +10 EM score.
I think it might be due to some weird characters that are removed in the preprocessing step.
The remaining errors are caused by the answer being part of a word like: if the answer was “japan” but the context had “Japanese” . In the BERT squad preprocessing they apply a span improve function https://github.com/aub-mind/arabert/blob/master/arabert/run_squad.py#L559 6
This has been discussed briefly here https://github.com/huggingface/transformers/issues?q=_improve_answer_span 12
Here is the link for the colab that I experimented on https://colab.research.google.com/drive/1UWwQkf_xBzkoLzPfB2GRs8OPR1Wtvz1Q#scrollTo=FU-YMtp_t0ec 11