The Marquis married firstly at London in 1828 Maria Garrow Birkett (who died 1831, aged 24), daughter of the late Daniel Birkett.[2] He married secondly in 1845 the widow of a Boston banker, Adeline Fowle (Mrs Samuel Welles), who died in 1869;[3] he married thirdly, in 1871, Georgiana-Gabrielle de Flahaut, marquise douairière de La Valette, who died in 1907; she was sister of Emily, Dowager Marchioness of Lansdowne.