It was rarely used as a residence, although Edward the Black Prince stayed at the castle in 1354 and 1365. The prince used these occasions to gather his feudal subjects at the castle in order for them to pay him homage. After the loss of Gascony, one of the key possessions of the Duchy, the contents of the castle were stripped out and removed to other residences. With an absent lord, the stewardship of the castle became much sought after as a result, and the castle and its estate became known for its efficient administration. The castle is recorded as having fallen into disrepair in a 1337 survey of the possessions of the Duchy of Cornwall. It was extensively repaired by order of the Black Prince but declined again following his death in 1376. By the time the antiquary John Leland saw it in the 16th century, it had fallen into ruin and had been extensively robbed for its stonework; as he put it, "the timber rooted up, the conduit pipes taken away, the roofe made sale of, the planchings rotten, the wals fallen down, and the hewed stones of the windowes, dournes, and clavels, pluct out to serve private buildings; onely there remayneth an utter defacement, to complayne upon this unregarded distresse." Henry VIII converted the castle's parkland back into ordinary countryside. With the castle out of use, a manor house was established during the 16th century a short distance away on lower-lying land adjoining the river. It is said to have been built on the site of a chapel dedicated to the Trinity that was destroyed during the English Reformation. Restormel Manor, now a grade II listed building, is still owned by the Duchy of Cornwall and is subdivided into luxury apartments with holiday accommodation in the outbuildings; the then Kate Middleton stayed there during the Christmas of 2009 and won a landmark victory over a paparazzo who photographed her there. Restormel has seen action only once during its long history, when a Parliamentary garrison occupied the crumbling ruins and made some basic repairs during the Civil War. It was invested by an opposing force loyal to Charles I, led by Sir Richard Grenville, a local member of the gentry who had been the member of Parliament for Fowey before the war. Grenville stormed the castle on 21 August 1644, whilst manoeuvring to encircle Parliamentary forces. It is not clear whether it was subsequently slighted but in an Parliamentary survey of 1649, it was recorded to be utterly ruined with only the outer walls still standing, and was deemed to be too badly ruined to be repaired and too worthless for there to be any value in demolishing it.