 Two of his great-uncles, Alexander Rives and William Cabell Rives, were Virginian lawyers and politicians.His mother died in 1849, and he was raised by his grandmother until his father remarried in 1853.He was educated by private tutors from 1851–56 and then at a school in Charlottesville, Virginia from 1856–60.