The MaskRCNN based models don’t support tracing due to the error you saw, but thanks to this PR 80 you can use torch.jit.script to compile the whole network.
You’ll need to get the most current version of torchvision by building it from source 32, then you can do
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
script_model = torch.jit.script(model)