To be on the same page, did you plot the masks as the output of your model, or to test your dataloader?
If you plotted the masks to check the result of your dataloader, then it means that their is either problem with your dataloader or the labels are wrong (check the json file of labels to verify the correctness of the labels).