Potter was born in Schenectady, New York on April 25, 1825. Potter was the eldest of seven children (six boys and one daughter) born to Sarah (née Nott) Potter (1799–1839) and Episcopalian Bishop Alonzo Potter (1800–1865) of Pennsylvania.