Could you print (some) values of the unique colors and check, if they might be noisy e.g. due to a resizing operation?
PS: It’s better to post code snippets by wrapping them into three backticks ```, as is makes debugging easier and also allows the search to index the code.