Hi Gautham,
in my opinion it is the best to install via whl file, otherwise it takes ages to be installed.
Might be a good idea to install pytorch >= 1.5.0, because you (already mentioned) torchscript. If it is not possible on raspi3 you can also go for flask and a wdsgi as backend.
A great pretrained transformer is the DETR object detector and you can already download it from facebooks github for pytorch.
Others like faster r-cnn might be a little to huge for a raspi3.
Hope it helps & have fun!