 She was the only daughter of Stephen Fox, 2nd Baron Holland (1745-1774), of Holland House, Kensington, (son of Henry Fox, 1st Baron Holland (1705-1774) by his wife Lady Caroline Lennox (1723-1774)) by his wife Lady Mary FitzPatrick, a daughter of John FitzPatrick, 1st Earl of Upper Ossory.Hon.Caroline Fox was the only sister of Henry Vassall-Fox, 3rd Baron Holland (1773-1840), of Holland House, who owned most of the land within the manor of Kensington, and was a niece of the Whig statesman Charles James Fox (1749-1806), who made Holland House a famous meeting place of prominent Whig politicians.