Colonel Hardenbergh bought James and Elizabeth Baumfree from slave traders and kept their family at his estate in a big hilly area called by the Dutch name Swartekill (just north of present-day Rifton), in the town of Esopus, New York, 95 miles north of New York City.