I agree with you, I am not just sure what to do. I am going to include my annotation here. I am using the Image(.jpg) and Mask(.png). The annotation has 38 classes + background. 

The Process I used to Prepare my Custom Datatset:

I annotated body parts of interests with lableme annotation tool.
I then generated a mask image with colors to indicate the parts of interests.(as seen in the colored image I uploaded)
I used the code from this link https://www.bulentsiyah.com/preprocessing-rgb-image-masks-to-segmentation-masks 3 to preprocess the mask image to label the mask image with the classes to get a new mask image(the dark one I uploaded here).

Ang then I used the new mask image(the dark one I added here) to train the model that is giving this error.
I am thinking that it’s because I did not include the original annotation in a bounding box? I mean the image with multi color.