The Van Cortlandts, a mercantile family prominent in New York affairs, established a grain plantation and grist mill on the property. The house was used during the Revolutionary War by the Comte de Rochambeau, Marquis de Lafayette, and George Washington. In 1889, the family sold the property to the City of New York as part of Van Cortlandt Park's creation.