 James Patton Anderson was born near Winchester in Franklin County, Tennessee.As a young boy, he moved with his family to Kentucky in 1831, where he lived for most of his childhood, and then to Mississippi in 1838.He attended the medical school of Jefferson College in Canonsburg, Pennsylvania in 1840, before a family financial crisis forced him to withdraw a short time before graduation in 1842.