The house was built in 1748 in the Georgian style by Africans enslaved by Frederick Van Cortlandt (1699–1749) for his family. Van Cortlandt died before its completion and the property was inherited by his son, James Van Cortlandt (1727–1787). It is a ​2 1⁄2-story, L-shaped house with a double hipped roof.