That is a rather inappropriate and silly comparison, to compare the color of a sweater, to political speech.  

If we lose the right to wear pink suspenders, how much do we lose?   If we lose the right to have political speech, how much is our democracy subverted and harmed?   I would argue a very great deal.