Originally part of the estate of Caleb Smith (1724–1800), the great grandson of Richard "Bull" Smith, the house was built in 1753 with his father Daniel Smith II. Although it has undergone many renovations, much of the original house still stands within the present building.[4] The house was moved in either 1955 or 1958 to a plot of land behind the Smithtown Main Public Library in the Village of the Branch, New York.[5]