Wearing NCR Armor in New Vegas... I spent a long time building up my Rep in New Vegas, Great Khans Like Me, Boomers Like Me.

When I put on NCR Armor my Rep Changes ( As to be expected ) but the biggest thing I'm noticing change is my NCR Rep.

It goes from liked to accepted..
Some of my other factions don't recognize me at all anymore.

Is there a way to wear NCR Armor and not be penalized? 