BufferedWriter takes most of the time
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 
 Speed and Memory Benchmarks: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
english = pipeline(
    "question-answering",
    model="distilbert-base-uncased-distilled-squad",
    tokenizer="distilbert-base-uncased-distilled-squad"
)

text1 = """It comes as pubs, bars, restaurants and other hospitality venues in England are told they must have a 22:00 closing time from Thursday.

Full details will be set out by the prime minister in Parliament later.

Boris Johnson is meeting the first ministers of Scotland, Wales and Northern Ireland and will address the nation in a live broadcast at 20:00 BST on Tuesday.

As well as the early closing time for hospitality venues, he is expected to announce they will be restricted by law to table service only.
"""

%%prun
english({'question': 'Which country is the news about?', 'context': text1})
```
The profiling result is 

```
6256 function calls (6155 primitive calls) in 1.097 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.713    0.713    0.713    0.713 {method 'write' of '_io.BufferedWriter' objects}
   37    0.229    0.006    0.229    0.006 {method 'matmul' of 'torch._C._TensorBase' objects}
   12    0.030    0.002    0.030    0.002 {built-in method matmul}
    5    0.020    0.004    0.020    0.004 {method 'dump' of '_pickle.Pickler' objects}
    6    0.019    0.003    0.019    0.003 {method 'softmax' of 'torch._C._TensorBase' objects}
   33    0.012    0.000    0.012    0.000 {method 'acquire' of '_thread.lock' objects}
    3    0.009    0.003    0.009    0.003 {built-in method posix.waitpid}
    6    0.009    0.002    0.009    0.002 {method 'masked_fill_' of 'torch._C._TensorBase' objects}
    6    0.009    0.001    0.009    0.001 {built-in method torch._C._nn.gelu}
   37    0.006    0.000    0.235    0.006 functional.py:1655(linear)
 94/1    0.005    0.000    0.325    0.325 module.py:710(_call_impl)
...
```


## Expected behavior

Most time is spent on inference such as the method `matmul`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
