Is the latest suggestion working for you?

For what I proposed, you would need to do instead
torchvision.models.detection.roi_heads.mask_rcnn_loss = My_Loss
i.e., directly modify what torchvision is using.
Is that what you tried doing?