inference over onnx output
# ❓ inference over onnx output

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
How to do decoding on the output obtained from onnx inference

I am trying to use onnx runtime for inferring on CPU by following the https://github.com/huggingface/transformers/blob/d822ab636b6a14ed50f7bca0797c1de42c19de61/notebooks/04-onnx-export.ipynb

I have a Marian MT hindi to english fine tuned [model ](https://huggingface.co/Helsinki-NLP/opus-mt-hi-en)which i have managed to convert using conver_graph_to_onnx.py script.
On calling `sequence, pooled = cpu_model.run(None, inputs_onnx)` 

I guess the pooled is the encoder output, and sequence is the final decoder output. Please correct if wrong

How can i use the api to get the final tokens (by greedy/beamsearch). For the normal way, we call the `generate` function. Is there any helper function to get the final decoded output form onnx output. Any other guidelines ?

Thanks!




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I have the same issue too! Please some guidelines ?<|||||>Same