Based on the example output, it looks like each channel contains different values.
Are you sure that the mask image does not contain a color code?
Could you try to call target.view(target.size(0), -1).unique(dim=1) and post the unique color values here?