 Her father was an assistant surgeon with the Confederate Army during the American Civil War while her mother was an aunt of the civil rights advocate Judge John Minor Wisdom.Witherspoon was orphaned by age 10 and raised at least in part by her older sister, Maud, who, while still in her teens founded the "Maud Witherspoon Rag Doll Manufacturing Company".Witherspoon's ancestors had reportedly once owned Ellington Plantation in St. Charles Parish, Louisiana.