You are right. Camvid masks have labels from 1 to 11. In my case, I do not have classes in order and they range from 0 to 23 and these are greyscale images. How to deal with this?
CamVid masks
Array Dimensions (360, 480)
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11], dtype=uint8)

My masks
Array Dimensions (512, 512, 3)
array([ 0,  2,  5,  9, 21, 22, 23], dtype=uint8)