This is from the mask_rcnn.py:
    During training, the model expects both the input tensors, as well as a targets dictionary,
    containing:
        - boxes (Tensor[N, 4]): the ground-truth boxes in [x0, y0, x1, y1] format, with values
          between 0 and H and 0 and W
        - labels (Tensor[N]): the class label for each ground-truth box
        - masks (Tensor[N, H, W]): the segmentation binary masks for each instance

here N is a ground-truth box, not class index.
So if I have (for example) 3 objects of class 1 and 2 objects of class2 and 1 object of class 3, image size 224x224, the mask array will be size (6x224x224), each map will be 0 for background and 1 for object. During training BCE is applied pixelwise, and the correct class is taken from the labels vector. Since for each object its class/label, bbox and mask are appended to a list, the order stays the same. so the correct class is taken from the labels vector.