Affluent summer vacationers and year-round empty nesters from Fairfield County and from other sections of Connecticut and the metropolitan New York region have been flocking to the area in the past three or four years .