Are u following this example? https://pytorch.org/docs/stable/torchvision/models.html#mask-r-cnn 39. If yes, I did the same last week and it worked fine to me. Please note the method def get_model_instance_segmentation(num_classes).
Are you adding new labels or you would like to have only your new labels? I did with only 6 classes (background + 5 classes). And it worked fine.