John Dillard's youngest son, James Dillard, purchased 1000 acres from Georgia state lottery holders and settled about 1823 in the northeast corner of Georgia in what would later become Rabun County, Georgia and the Town of Dillard.