The sale would only add to the seismic cultural shifts already under way in New York City and especially in Manhattan , where soaring housing costs have made the borough increasingly inhospitable to working-class and middle-class residents .