Unfortunately resizing first and then adding in the colored masks did not change anything (unless I did something wrong in these steps).
I printed the list of unique colors and the length and this is what I get:
print(colors)
[[  7. 133.   0.]
 [  8. 132.   0.]
 [  8. 132.   2.]
 ...
 [255. 254. 255.]
 [255. 255. 253.]
 [255. 255. 255.]]

len(colors) = 3809 but that is strange because it should be 5 (or maybe 6 if it counts the black background).