You can check the new run_qa script 39. It does need a special subclass of Trainer because the post-processing is fairly complex, but it uses compute_metrics and the squad metrics from the datasets library.