ONNX support for gpt-neox
### Feature request

Is there any possibility that we can convert gpt-neox to onnx using transformer library?

### Motivation

onnx model will be faster.

### Your contribution

-
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @hannan72 we have a guide on adding support to export new architectures to ONNX: https://huggingface.co/docs/transformers/serialization#contributing-a-new-configuration-to-transformers

Would you like to have a go at implementing the ONNX config for this model? It should be similar to what has already been done for other GPT models like `gpt-neo`<|||||>Hi @lewtun 
It is totally welcome if I can convert GPT like models to onnx. I did the conversion for gpt-j-6b. The problem was that the inference time improvement was only 5% in comparison to PyTorch on my GPU. Did you have any benchmark for the conversion of GPT models to onnx?<|||||>Hi @hannan72 

> It is totally welcome if I can convert GPT like models to onnx. I did the conversion for gpt-j-6b.

Cool, this would be a nice addition! 

> The problem was that the inference time improvement was only 5% in comparison to PyTorch on my GPU. Did you have any benchmark for the conversion of GPT models to onnx?

Were you running on ONNXRuntime? We've found that to really speed up generations with GPT-like models, one needs to include ONNXRuntime's [IO bindings](https://onnxruntime.ai/docs/api/python/api_summary.html#iobinding) and [ORTValue](https://onnxruntime.ai/docs/api/python/api_summary.html#ortvalue). I don't have a benchmark for this currently, but it's something we're looking into in `optimum`
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.