white was born on may 23, 1822 in new york city. he was born to richard mansfield white (1797–1849) and ann eliza (née tousey) white (1802–1842), and was eight in descent from john white, a puritan who was one of the founders of cambridge, massachusetts and hartford, connecticut. his father, a shipping and commission merchant, was from a wealthy old new england family that lived in new york city. the elder white also served as secretary of the allaire iron works company.