Hello,
I trained frcnn model with automatic mixed precision and exported it to ONNX. I wonder however how would inference look like programmaticaly to leverage the speed up of mixed precision model, since pytorch uses with autocast():, and I can’t come with an idea how to put it in the inference engine, like onnxruntime.
My specs:
torch==1.6.0+cu101
torchvision==0.7.0+cu101
onnx==1.7.0
onnxruntime-gpu==1.4.0
Model exports just fine:
torch.onnx.export(model, 
                  x, 
                  "model_16.onnx", 
                  verbose=True, do_constant_folding=True, opset_version=12,
                  input_names=input_names, output_names=output_names)

But I wonder how to leverage mixed precision speed up here:

import onnxruntime as ort
ort_session = ort.InferenceSession('model_16.onnx')
outputs = ort_session.run(None, {'input': x.numpy()})