[Benchmark] Jetson Nano DistillBERT SQuAD benchmark 
# 🖥 Benchmarking `transformers`

## Benchmark

DistillBERT SQuAD

## Set-up

- Jetson Nano
- Jetpack 4.4
- onnxruntime 1.13.1

Benchmark script https://gist.github.com/arijitx/1400d3d4e07fc517d6c5bfea506c2353

## Results

tokens/sec

|                    | Pytorch-GPU | Pytorch-CPU(4 cores) | onnx-CPU (4 core) | onnx-CUDA | onnx-TRT |
|--------------------|-------------|----------------------|-------------------|-----------|----------|
| Distill BERT SQuAD |         570 |                   61 |               107 |  605      |     fail     |


For tensorrt you need to manage dynamic axis with a script in the nuphar folder of onnx runtime repo (`symbolic_shape_infer.py`).
It gives best result for small batch (like single example).
It requires some customization in parameters because default one are not that good (check the Nvidia doc for the workspace size, etc.)
Of course, it requires a specific compilation but I suppose you did it.
For CPU you may want to install the onnx runtime without gpu support so you get openmp.<|||||>Ya I did `symbolic_shape_infer.py` but still for some reason, the model is not running with tensorrtProvider, it starts consuming memory and then crashes after the memory is loaded, I did try with workspace_size , iteration as mentioned in the  [Onnx TensorRT Provider doc](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/TensorRT-ExecutionProvider.md) anything `ORT_TENSORRT_MAX_PARTITION_ITERATIONS  > 1 ` crashes. 

Still Jetpack 4.4 is not officially supported by onnx, and I couldn't find a way to export pytorch models with LongTensors directly to TensorRT engines :( 

Am able to run successfully in the CUDAProvider, CPU can be optimized further with a openmp specific build.<|||||>My own XP with onnx has been painful too :-) 
May be you will be interested in this article https://medium.com/@fanzongshaoxing/accelerate-pytorch-model-with-tensorrt-via-onnx-d5b5164b369 ?
It s about tensorrt on Python without onnx runtime (didn't try yet but was curious)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
