One of the early owners of the property was Alexander Slidell Mackenzie, who lived there from 1840 to 1848. Edwin Bartlett obtained the property and build Rockwood, an English Gothic castle of locally quarried stone. Bartlett sold the house to his business partner William Henry Aspinwall in 1860; Aspinwall made it his summer home and improved the property and house, and purchased enough land to make his estate 200 acres (81 ha).