I did another test of converting the fasterrcnn_mobilenet_v3_large_320_fpn pretrained model into a torchscript module. The reproducible code is available here.
from torchvision.models.detection.faster_rcnn import fasterrcnn_mobilenet_v3_large_320_fpn

model = fasterrcnn_mobilenet_v3_large_320_fpn(pretrained=True)
model.eval()
image = cv2.imread("./test1.png")
image = cv2.resize(image, dsize=(320, 320))
transform = torchvision.transforms.Compose([
     torchvision.transforms.ToTensor()
])
image = transform(image)
script_model = torch.jit.script(model)
torch.jit.save(script_model, "scripted_model_mobilenetv3_large_320_fpn.pt")
print(script_model([image]))

Python Output:
({}, [{'boxes': tensor([[ 57.2329,  61.6968, 240.5976, 270.2655],
        [ 49.8475, 110.0096, 137.2613, 300.0203],
        [191.1161,  39.0309, 286.9027,  94.8667],
        [188.2992,  36.5230, 288.7202,  97.4937],
        [ 53.7912,  65.7180, 106.5928, 123.5672],
        [ 54.1852, 112.7328, 134.0000, 302.0532],
        [ 30.1234,  80.8505,  97.9023, 221.6472],
        [ 27.9103,  47.1386,  42.7512,  66.0757],
        [ 50.6809,  45.9808, 110.9343, 113.8588],
        [166.8281,  51.4230, 193.9323,  72.8806],
        [  1.9139,   1.3991,  28.6922, 141.1715],
        [ 34.4260,  77.5499, 105.4713, 233.3638],
        [ 28.4753,  50.9322,  48.0249,  68.2570],
        [158.8731,  53.8971, 182.2708,  70.5178]], grad_fn=<StackBackward>), 'labels': tensor([ 2, 18,  3,  8,  2, 17, 62,  3, 64,  3, 72,  2,  4,  3]), 'scores': tensor([0.9982, 0.7730, 0.6875, 0.3241, 0.2950, 0.1868, 0.1649, 0.0912, 0.0715,
        0.0705, 0.0608, 0.0604, 0.0557, 0.0547], grad_fn=<IndexBackward>)}])

Using the C++ code above:
C++ Output
{boxes:    0.0000    0.0000  320.0000  320.0000
  36.9277    2.1780  264.9876  307.3490
  11.3890    0.0000  319.6223  316.5984
[ CPUFloatType{3,4} ], labels:  72
  1
 82
[ CPULongType{3} ], scores:  0.1217
 0.0593
 0.0533
[ CPUFloatType{3} ]}

test1 image from D2Go repository 1