If your target is an RGB image containing the color map, you would have to map it to class indices first.
I’m not familiar with the NYUv2 dataset, but maybe you’ll find some conversion between color codes to class indices.
If you can’t find it, you could create it by yourself using a dict.