This included paintings by Rubens, Lawrence and Reynolds. The principal seat of the Seymour family had been Maiden Bradley in Wiltshire, but for one more generation it remained Stover. The Stover purchase included the Stover Canal and the Haytor quarries and Haytor Granite Tramway. He added a large porte cochere with Doric columns to Stover House and built a matching entrance lodge. Somerset married twice, firstly on 24 June 1800 to Lady Charlotte Douglas-Hamilton (6 April 1772 – Somerset House, Park Lane, London, 10 June 1827), daughter of Archibald Hamilton, 9th Duke of Hamilton, by whom he had seven children:  Following his first wife's death in 1827 he remarried on 28 July 1836 at Marylebone, Portland Place, London, to Margaret Shaw-Stewart, daughter of Sir Michael Shaw-Stewart, 5th Baronet of Blackhall, Renfrewshire, by his wife Catherine Maxwell, daughter of Sir William Maxwell, 3rd Baronet. The marriage was childless. Somerset died at Somerset House in London, in August 1855, aged 80, and was buried at Kensal Green Cemetery, London. Margaret, his second wife, died at Somerset House on 18 July 1880, and was buried with her husband.