anderson was married twice. in 1850 he wed sarah gibson, and the couple had two children together, a son and a daughter; after her death he married martha mellette on december 24, 1874. five years later anderson died at the age of 57 in beaufort, south carolina, and is buried there in the churchyard cemetery of st. helena's episcopal church.