The house was owned by James Barbour (brother of the railroad's president John S. Barbour Jr.) during the Civil War but the Barbour family's main residence was on a ridge about 1 miles to the south (and during the war was used by Confederate General J.E.B. Stuart).