Pytorch Vs Onnx: Pytorch is faster and provides different output
## Environment info

- `transformers` version: 3.5.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

albert, bert, GPT2, XLM: @LysandreJik 
Onnx: @mfuntowicz


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1UwgWgUF4k_GPJ5TcziHo4eH_rRFQeNVL?usp=sharing


## Expected behavior

I have followed the  the onnx export tutorial:
https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb
However, I have found 2 issues:
1. Pytorch is faster than Onnx.
2. Onnx produce different embedding output than Pytorch.

Could anyone help me to figure out the issue ?
I also got different results 🤦‍<|||||>I have figured out the problem, but I don't have the solution.
When you use a single sample per batch it works correctly, but when you use more than one sample per batch, the results are totally different.<|||||>@mfuntowicz and @LysandreJik , it will be great if you could show us an example for how to correctly use batch processing for onnx inference.<|||||>Hey @agemagician! I have exactly the same problem, great inference time with onnx when batch size = 1 but when batch size is increased, raw pytorch wins over onnx. Did you find any solution for the issue?<|||||>unfortunately, not.<|||||>That's unfortunate..<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.