Based on the output it seems that some colors (e.g. white, which is given as [255, 255, 255]) have some “neightbors” with a small difference (e.g. [255, 254, 255]), which could be introduced by an interpolation method in e.g. resizing.
If you are resizing the image, I would recommend to use the nearest neighbor interpolation method, as it would keep the pixel values without adding new unique ones.
In case you cannot isolate the operation, which is adding these values, you could try to cluster the values afterwards into the desired 6 classes.