 Phelan was born in Aberdeen, Mississippi to James Phelan Sr., a member of the Confederate Congress, and Eliza J Phelan.He was of Irish descent on his father's side, and Scottish descent on his mother' side.He attended various schools.