Solved by sgugger in post #2 
              
              
                The evaluation for question-answering is pretty long as the post-processing (going from the predictions of the models to spans of texts in the contexts) is not on the GPU and is rather long. If it’s for an evaluation during training, you should use a smaller validation set.