After his mother's death in April 1839 from complications arising from giving birth to their seventh child (and only daughter), his father remarried in 1840 (to his mother's cousin Sara Benedict) and had three more children, all boys. Sara died in 1864 and his father remarried for the third time to Frances Seton, just three months before his death in July 1865. Among his siblings were brothers Howard Potter, a New York City banker, Robert Brown Potter, a General in the American Civil War (to whom he gifted a house known as "The Rocks" in Newport, Rhode Island),[2] Edward Tuckerman Potter, an architect who designed the Nott Memorial at Union College, Henry Codman Potter, who succeeded Horatio Potter as Bishop of New York in 1887, Eliphalet Nott Potter, who served as President of Union College and Hobart College, and William Appleton Potter (1842–1909), also an architect who designed the Church of the Presidents in Elberon, New Jersey.[3]