That what these historic designation are all about.  You can't build affordable housing that fits with the character of these neighborhoods, it would be too expensive.  These designation strip the rights of the property owners and puts the control into the hands of the historic district and SHPO.  It stop development, and reduces any chance of affordable housing showing up in these areas.