 Handy was born in Florence, Alabama, the son of Elizabeth Brewer and Charles Barnard Handy.His father was the pastor of a small church in Guntersville, a small town in northeast central Alabama.Handy wrote in his 1941 autobiography, "Father of the Blues", that he was born in a log cabin built by his grandfather William Wise Handy, who became an African Methodist Episcopal minister after the Emancipation Proclamation.