Onnx + TensorRT uses CPU not GPU
## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@mfuntowicz 

## Information

Model I am using Bert:

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1mqn99U2NMm-fdw213h0xsqm_uRwRkNzp?usp=sharing

## Expected behavior

I am trying to run onnx using tensorrt on the gpu, but it always uses the CPU.
I tried to install different version or onnx and cuda with not luck.
Any idea why "TensorrtExecutionProvider" runs on the CPU rather than GPU ? and how to fix it ?

@mfuntowicz - sorry to ping you here :D Do you have a "high level" idea why this might not work?<|||||>@agemagician IIUC TensorRT needs a GPU with Tensor cores to work correctly. You're using a P100 in the notebook which doesn't have those. So Onnx will fall back to using CUDAExecutionProvider. Now that itself may be slow because of [this](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Perf_Tuning.md#why-is-my-model-running-slower-on-gpu-than-cpu). I'd suggest setting verbose logging level and checking. To enable set the following:
```
options = SessionOptions()
options.log_verbosity_level=2
options.log_severity_level=0
```
Then in the console logs if you see something like
`2020-10-16 20:25:31.386914373 [W:onnxruntime:Default, fallback_cpu_capability.h:140 GetCpuPreferedNodes] Force fallback to CPU execution for node: Gather_9`, or there's a huge list
```
2020-10-16 20:25:31.406073394 [V:onnxruntime:, inference_session.cc:869 TransformGraph]  Provider: [CPUExecutionProvider]: [Gather (Gather_9), Unsqueeze (Unsqueeze_10), DynamicQuantizeLinear (237_QuantizeLinear), QAttention (Attention_qua...
```
 that indicates that some parts of your model are not GPU compatible because Onnx doesn't have the supported operators.

In my case it was happening because I was using a quantized model with the CUDA provider, but the quantized model needs TensorRT to run I believe.<|||||>Thanks @bdalal  for your reply. I have tested it also on V100 but I had the same issue.
Anyway, I think the benefits vs troubles, seems to be not worth it for now.
I hope Nvidia, onnx and Pytorch provide a better integration on the future for tensor rt.
