How to speed up inference step in BertQuestionAnswering?
I'm working on a QA system which uses a pre-trained BertQA model. At this point, even if I use GPU, the step generating start_scores and end_scores  for a set of 20 candidate passages till takes a few seconds which is the bottleneck of my application.

I just wonder do we have any strategies/tricks to speed up this step? So far, it seems using multiple GPUs at the inference step does not help at all. Any advice greatly appreciated!
To improve inference speed you can use ONNX (also see here: https://github.com/huggingface/transformers/issues/260). In addition, you can opt for a distilled model rather than the full model. <|||||>Hi! I've used this:


distilbert-base-uncased-distilled-squad |  
-- | --
or 

distilbert-base-cased-distilled-squad |  
-- | --

It improved quite a bit! 
<|||||>@ZordoC how much speed improvement did you observe?