 Rabun was born in Halifax County in the Province of North Carolina and moved with his parents to current-day Hancock County, Georgia (then a part of Greene County) in 1785.Rabun's home in Powellton is located ten miles northeast of Sparta.In 1793 he married Mary Battle, and the couple had one son and six daughters.