In 1851, Potter was married to Virginia Mitchell (1827–1890) of Philadelphia, the daughter of Matthew and Louisa (née Kidd) Mitchell. The Potters owned a country house, known as "Nutwood", two miles from Trinity Church in New Rochelle, overlooking the Long Island Sound.[1] Together, they were the parents of:[4]