 Ancaster married twice.His first wife, whom he married on 11 November 1762 in Cork Street in London, was Harriot Pitt (1745–1763), the only daughter and heiress of George Morton Pitt.After the death of his first wife on 23 April 1763, he remarried to Mary Anne Layard (1733–1804), a daughter of Maj Peter Layard of Sutton Friars, on 2 January 1769 in St James's.