on returning to britain, fitzpatrick embarked on an affair with anne fitzroy, wife of prime minister augustus fitzroy, 3rd duke of grafton, (and daughter of henry liddell, 1st baron ravensworth). this resulted in the prime minister having an affair with a courtesan, the birth of a child, the duchess's divorce and her marriage to fitzpatrick in 1769. like his younger brother richard, fitzpatrick was a friend of charles james fox, whilst their half-sister mary fitzpatrick had married charles james fox's older brother stephen fox, 2nd baron holland. as such fitzpatrick became step-father to henry vassall-fox, 3rd baron holland on the death of his parents. he was also a noted friend to horace walpole; a large volume of correspondence between walpole and fitzpatrick's wife, the countess of upper ossory survives.