what you told is correct ,actually all the white color in the image is ignore index. Sky, building and host vehicle all should have corresponding classes but the dataset provided by them labels some parts as 255.
Is their any workaround?