From 1366, the ecclesial control of Abdinghof over a chapel at the Externsteine is well documented. This lasted into the 17th century. It involved a hermitage at this location, with individuals named as hermits in 1385 and 1469. The document from 1385 also mentions an "Upper Altar", likely referring to the altar in the "Höhenkammer". There remains a contradiction between the use of the Externsteine as a simple roadhouse for travellers and as an hermitage on the one hand and the presence of the monumental relief and the Arcosolium on the other. These may indeed have been a reconstruction of the Holy Sepulchre. The remains visible today indicate the possibility that the Externsteine were intended as a destination for pilgrims unable to travel to Jerusalem. To that end many medieval churches created . However, it is unclear who could have built it here and why there is no written evidence of such a relatively elaborate undertaking. The site was within the County of Lippe, formerly a county within the Duchy of Saxony, which gained imperial immediacy by 1413, throughout the Early modern period. The hermitage apparently became a hideout for bandits and in the early 16th century was dissolved by the rulers. This was roughly at the time that the Reformation was introduced in Lippe (1538) and church activity at the site ceased. The original claims associating the Externsteine with Saxon pagan worship were made by Hermann Hamelmann, who in his "Delineatio Oppidorum Westfaliae" (1564) claimed to take the information from older authorities (which cannot now be recovered or identified),  Around 1592, the Count's local forest warden lived in the caves. Circa 1660, Herman Adolph, Count of Lippe designed the area for use as a hunting lodge and "Lustschloss". This was a fortress of two squat round towers flanking a central gatehouse, built against the eastern side of the Externsteine. The excavated foundations show relatively thin walls, indicating that these walls were never intended as fortifications but were just ornamental. A stairwell next to rock I gave access to a viewing platform on top. Likely associated with this era is the bearded figure on the platform and the coat of arms inserted into rock IV. The structure was rarely used, and fell into dilapidation in the 18th century.