I am not a big expert in this area but just for letting know what I have discovered.
Converting  PyTorch models to TorchScripts models is a good idea. You get a file that contains the arch class and weights. In the first run of the model gets optimized and after that is pretty fast. I am getting around 1.98 s ± 42.1 ms in my Ryzen 1700 with Mask-RCNN Resnet 50 (the implementation of torchvision).
I have also discovered quantization 31 . It optimizes models and make them able to run on a mobile or arm devices like Raspberry. However, after quantization the model losses some precision. Here you have an image explaining the process:
image1396×1574 113 KB
You could run this models on your computer not just mobile devices