The earthwork fortifications of the castle were rebuilt in stone by his successor, Bernard de Balliol I during the latter half of the 12th century, giving rise to the town's name.[6] The castle passed down through the Balliol family (of which the Scottish king, John Balliol, was the most important member) and then into the possession of Richard Neville, Earl of Warwick. King Richard III inherited it through his wife, Anne Neville, but it fell into ruins in the century after his death.