Performance Issue in doing inferencing hugging face models
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:3.2.0
- Platform:windows azure with GPU
- Python version: 3.6
- PyTorch version (GPU?):CUDA 10.1
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

We have created a flask end point to cater to multiple ML models such as Longformer Question answeing,zero shot learning, paraphrasing.But when these functions are called from flask end point the CPU percentage utilisation increases to more than 97% even though the GPU is already in place. Any idea why CPU takes a hit as this is just used for inferencing purpose. 
Also is there a way to check the memory size allocated to a variable when we load the torch pretrained model in the variable and pass this as an argument in the function. If there are concurrent calls does this memory requirement increases or is the same memory is being referred.
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!