Hi,
thank you for sharing your problem!
Naïvely, I would expect it may be better to train the model only to discriminate between exclusive options if the labels allow that. So from your example, you would have [blue, red, green], [square-shaped, curve-shaped], [car, bike].
So in this view, the cross-entropy loss would penalise red and green and favour blue for images labelled blue, but leave the probabilities for unrelated labels alone. You didn’t say, but for this I would expect exclusive categories (and only those) to “participate” in a mutual softmax or so.
It’ll be very interesting to hear what works and what does not.
Best regards
Thomas