 James was the oldest of three boys—his brothers were Robert Gamble Cabell III (1881–1968) and John Lottier Cabell (1883–1946).His parents separated and were later divorced in 1907.His aunt was the suffragist and educationist Mary-Cooke Branch Munford.