As the error message says, the model is using PyTorch 0.1.2-style Functions. (Newer autograd.Functions cannot be traced either, but that is another story…)
You probably want to convert the model to use the nms functions provided by TorchVision, those have tracing support.
Best regards
Thomas