This may be happening if pretrained_backbone=True and the model cannot find the official pretrained weights in the usual path. If you just want to use your own weights you can set pretrained_backbone to False and just load your weights manually like this
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained_backbone=False)
model.load_state_dict(torch.load(PATH))