Gantt was born to a slave-owning plantation family in Calvert County, Maryland at the outbreak of the American Civil War. When the war ended the family lost their slaves and land, and moved to Baltimore.[1]