 The Gresham family moved to Holt in the fifteenth century.According to Francis Blomefield in "An essay towards a topographical history of the county of Norfolk" (1808), James Gresham, the grandfather of Sir John Gresham, was "the son of John Gresham, Gent., of Gresham".The remains of a fortified house called Gresham Castle are near the village, opposite the Chequers Pub.