The rapid escalation of home prices since 2004 has reduced the number of Connecticut towns with '' affordable '' median home prices from 67 to just 12 -- and none were in Fairfield or New Haven Counties .