After I use the uer py training model to convert the transformers model, the decoding speed is not very fast. I want to improve the speed by converting the model onnx.

python -m transformers.onnx --model=bert-base-cased onnx/bert-base-cased/
Some problems were encountered using the converted model.
According to the official documentation, there is no generation method.

I checked the case list of gpt2 conversion onnx. The gpt2 model can be decoded normally and the speed has been improved.


  

      github.com
  

  
    microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/notebooks/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb


    {
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference PyTorch GPT2 Model with ONNX Runtime on CPU\n",
    "\n",
    "In this tutorial, you'll be introduced to how to load a GPT2 model from PyTorch, convert it to ONNX, and inference it using ONNX Runtime using IO Binding. Note that past state is used to get better performance."
   ]
  },
  {



  This file has been truncated. show original

  

  
    
    
  

  



image1192×134 1.05 KB

An exception occurred after replacing with the converted Chinese model.

image1310×257 4.52 KB

Hope to get help here, thank you!