It was built by Francis Stott of Stottville, New York, at the suggestion of Dr. Thomas C. Durant. Durant and Mrs. Stott's family had briefly been in business together as Durant, Lathrop & Co. in Albany, New York, during the 1840s. When Bluff Point was sold to magazine publisher Robert Collier in 1905, most of the original structures were modified and expanded into the present score of buildings — the huge walk-in fireplace, the bowling alley, the bridge to the gazebo, etc.