Not in the least.  Called 'freedom of association'.  You can't choose to live where you can afford to live.  You can't stop others from living there if they can afford to.  BUT, if you own the land in an area and don't want to sell to 'em, you don't have to.  Apparently civics is no longer taught...