Reviving this topic as I have come to a similar conclusion. I have been testing and a variety of things but for a Resnet50 network with a large input size (3MP image), and with an RTX3070, I have come to the conclusion that tensorRT engine with fp32 is actually slower than cudnn inference on pytorch with JIT trace + cuda AMP:
Time for total prediction pytorch JIT = 0.06634163856506348
Time for total prediction trt = 0.07124924659729004
It is the same model which was exported from pytorch to ONNX and then converted from ONNX to trt. I have also tested torch2trt and TRTorch and so far only TRTorch appear to show some small gain in speed but both are still alpha projects with some problems with output not matching the original model.