In 1482, the castle was burned down and the walls were torn down, except for the parts that did not have a military function. These parts probably were incorporated into the castle when it was rebuilt during the early 16th century. The castle is mentioned in an inventory of the possessions of Steven van Zuylen from 1506, and again in a list of fiefdoms in the province Utrecht from 1536.