Sorry for not being clear on the description. I meant that you would have to make sure to use a target tensor, which would contain all possible classes, to create the mapping.
E.g. in case you are only checking the unique colors of a single sample, this target tensor could potentially only contain a class subset of all available classes in the dataset.
In that case, you can stack the target tensors to create a “dataset target” tensor and check the unique colors on it.