Henry Grattan called him the most honourable and humane judge he ever knew. Barrington's low opinion of Kelly as a judge was fully shared by the English-born politician Edward Cooke, always a stern critic of the Irish judiciary: he wrote that "Kelly has been most unfortunate in his judgments: (there is) scarcely one upon a dubious point which has not been set aside." He had a town house in Dublin and a country seat, Kellyville (formerly Derrinroe), near Ballintubbert, County Laois, which he purchased around 1777, and substantially rebuilt. He retired in 1801: by some accounts he resigned in protest against the passing of the Act of Union 1800, which destroyed the independent Irish Parliament to which Kelly was devoted. He died in Dublin in 1809. He married Frances Hickie, daughter of James Jephson Hickie of Carrick-on-Suir and his second wife Anne Salisbury Jephson. They had three daughters, Annabella, who married Sir George Pigott, first of the Pigott baronets, Harriet, who married Sir Richard Bligh St George, the second of the St George baronets, and Charlotte, and one son, Thomas Kelly (1769–1855).