Besides, he said, giving every boy in Virginia three years of school would have limited practical utility, would deprive farm families of much-needed labor power, and would leave the typical "scholar" unfitted for the return to hard labor that awaited him. When James Barbour left the Senate in 1825, Giles attempted to persuade the legislature to appoint him as replacement; they appointed John Randolph instead. In 1826, Giles was again elected to the House of Delegates, and in 1827 he was elected Governor; Giles served as Governor of Virginia for three terms, from March 4, 1827 to March 4, 1830. From the governorship, Giles encouraged Virginia's Senator Littleton Waller Tazewell to organize a southern resistance to the American System of Henry Clay centered on a boycott on northern manufactures. Tazewell found little support for it among southern senators. In Giles's last term, he was a member of the Virginia Constitutional Convention of 1829–1830 where he strongly supported the existing apportionment of the House of Delegates giving the eastern counties of Virginia, with a minority of the voters, control of the legislature. He did favor reform of the suffrage requirements, however. Giles also opposed the movement in the Convention to strengthen his own office, the governorship. Strong governorships in other states, such as New York, were at the center of political machines kept together by patronage and corruption, he said, and the reason that Virginia had not suffered from those ills was that the governorship in his state was too weak to be worth fighting for. Rather than follow the example of New York, with its party machine, it was better for Virginia to retain George Mason's executive model. Giles lost to some extent: while the governor's term remained short and was still accountable to the General Assembly, the Constitution of 1830 abolished the privy council, thus making the governorship a bit more independent. Giles married twice; first, Martha Peyton Tabb, in 1797; he built his 28-room house, "The Wigwam," for her. They had three children. After she died in 1808, he married Frances Ann Gwynn in 1810 and had an additional three children. Counties in two states were named in his honor: Giles County, Virginia and Giles County, Tennessee. His name also graces a residence hall at the College of William and Mary. The Wigwam was added to the National Register of Historic Places in 1969.