A question on twitter

can’t we import T5Model from transformer and then may be pass it through a wrapper or decorator for onnx optimization. Or do we really need to change some ops in torch so that they can be onnx compatible?

We could export the T5 as it is but then we can’t use past_key_values because they will be None for first decoding step and onnx doesn’t accept None input.  So here we are exporting individual components to be able to use past_key_values
Here you could also export the encoder and lm_head directly instead of creating classes for it. I did it this way to make it more readable. And we need to write the extra code in OnnxT5 class to make it compatible with generate.
Feel free to suggest changes if you think it can be improved