 The building was extended by in 1804 by Daniel Crommelin Verplanck, the grandson of Gulian Verplanck II, who also laid out the garden.When Marquis de Lafayette visited the house on his return to America in 1824, he stayed in the new addition.James Brown was born into slavery in Fredericktown, Maryland in 1793, and escaped via the Underground Railroad to New York City, where found work as a waiter at the Verplanck's mansion on Wall St. A story handed down in the Verplanck family relates that a dinner guest recognized Brown as an escaped slave, and notified the owner in Maryland.