Umm…
I think this issue is from architecture.
Do you understand Mask rcnn how to work?
############################################################
#  Feature Pyramid Network Heads
############################################################
x = x.view(-1, 1024)
        mrcnn_class_logits = self.linear_class(x)
        mrcnn_probs = self.softmax(mrcnn_class_logits)

All of  mrcnn_probability are background.
def refine_detections(rois, probs, deltas, window, config):
# Class IDs per ROI
    _, class_ids = torch.max(probs, dim=1)

The background class_id is 0
So input to next step, can not  filter out background.
Thanks again for your support.  I feel helpless.