 He was born the only son of William Fitzwilliam, 4th Earl Fitzwilliam and his first wife, Lady Charlotte Ponsonby.He was a pupil at Eton College from 1796 to 1802.Before inheriting the Earldom on 8 February 1833 on the death of his father, he was known by the courtesy title of Viscount Milton.