The script has been tested on the squad and squad v2 datasets and gives the same results (roughly) as the previous one. The reason might be linked to the fast tokenizers offset mappings not really working with Arabic, maybe? This would be the thing I would check first.
Maybe you could try to replicate the steps in this notebook 23 on your dataset to check all the preprocessing looks correct?