[Benchmark] Pipeline for question answering
# 🖥 Benchmarking `transformers`

## Benchmark

I'm trying to benchmark QA model with `bert-large-uncased-whole-word-masking-finetuned-squad`. But it seems it is extremely slow e.g. 3-4 sec for 1 question with 2 contexts.

I feel there is something I'm missing in pipeline.

## Sample Code:
```
def answer(self, contexts:List[str], question:str, **kwargs):
   ## tokenizer, model, pipeline all are cached in actual implementation 
  ## via [reify](https://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html) 
  ## so model loading is not a problem.
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_len=500)
    model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking- 
        finetuned-squad')
    pipeline('question-answering', model=model, tokenizer=tokenizer)
    pipeline_input = []
    for c in contexts:
        pipeline_input.append({
            'question' : question,
            'context' : c
        })
    answers = pipeline(pipeline_input)
```

## Set-up
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
memory: 16GB


same experience with a 2080TI. It s like it s not batched...<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>As @pommedeterresautee mentioned, the samples are not batched. I've made a [simple modification](https://gist.github.com/vinicius-cleves/1b2d79a9665d165ac22451c225b2f8b0) to do batch inference on pytorch, but it didn`t seem to help much with the total processing time. 
