once released, hubert regained his grip on power, becoming the royal justiciar and being made the earl of kent, before finally recovering the three castles in 1219 during the reign of king henry iii. hubert fell from power in 1232 and was stripped of the castles, which were placed under the command of walerund teutonicus, a royal servant; having been reconciled with the king in 1234, the castles were briefly returned to hubert, but he fell out with king henry iii again in 1239 and they were taken back and assigned to walerund. walerund built a new hall, buttery and pantry at the castle in 1244. in 1254, white castle and its sister fortifications were granted to king henry's eldest son, and later king, prince edward.