Then refugees from Fidel Castro 's Cuba sought work in its dying factories , and later immigrants from South America crammed its Victorian houses as boarders so they could work tending lawns in nearby Greenwich , Conn. , and Rye , N.Y . The village 's proportion of Hispanics soared to 46 percent in 2000 from 16 percent in 1980 .