 His paternal grandfather, also named Alexander Brown, was born near Perth, Scotland and came to Virginia in 1811, where he married Virginian Lucy Shandes Rives.On his mother's side, he came from one of Virginia's oldest families, the Cabells, who hailed from Buckinghamshire.He was descended from American Revolution officer Col. William Cabell.