It is believed that blacks first settled in Fair Haven because land was available there at reasonable prices and was near Rumson , where many of them worked on the estates of some of New Jersey 's wealthiest families .