Hello,
thanks for the suggestions. I run the onnx runtime profiler on 16FP model from torch, but I’m not exactly sure how to look for execution of FP16 operations there. I’m attaching the log.
http://www.mediafire.com/file/rel0ze3y963nohv/onnxruntime_profile__2020-08-28_10-35-51.json/file 7
Here’s the code I used to run the profiler:
import onnxruntime as ort
options = ort.SessionOptions()
options.enable_profiling = True
ort_session = ort.InferenceSession('model_16.onnx', options)
outputs = ort_session.run(None, {'input': images[0].cpu().numpy()})
prof_file = ort_session.end_profiling()

Anyway, if I do simple time measurement for inference, I don’t see much difference. To be honest I expected ONNX model to run faster.
1.Pure torch 16FP model:
from imutils.video import FPS
fps = FPS().start()
for i in range(100):
    images = list(image.to('cuda:0') for image in x)
    with autocast():
        pred = model(images)
    
    fps.update()

fps.stop()
print('Time taken: {:.2f}'.format(fps.elapsed()))
print('~ FPS : {:.2f}'.format(fps.fps()))

Time taken: 2.19
~ FPS : 45.57


Torch->ONNX 16FP model:

import onnxruntime as ort
ort_session = ort.InferenceSession('model_16.onnx')
fps = FPS().start()

for i in range(100):
    outputs = ort_session.run(None, {'input': images[0].cpu().numpy()})
    fps.update()

fps.stop()
print('Time taken: {:.2f}'.format(fps.elapsed()))
print('~ FPS : {:.2f}'.format(fps.fps()))

Time taken: 2.15
~ FPS : 46.61