Since then the castle has been in Welf hands uninterruptedly for 708 years until the demise of the Kingdom of Hanover in 1866. Empress Maria of Brabant, widow of Emperor Otto IV, made out a deed at the "Hertsberg" in 1218, thus the castle was an imperial residence for a short while. In 1279 the castle served as the residence for the widow of Duke Albert the Great. From 1337 to 1714 the castle was a ducal residence almost continuously. In 1290 the castle became the residence of the Welf line of Brunswick-Grubenhagen, which was formed around this time. From 1486 the dukes of the Principality of Grubenhagen lived here until they died out in 1596. Thereafter the estate was transferred to the Welf line of Brunswick-Lüneburg. There was a major fire at the castle in 1510. The lord of the castle, Duke Philip I, his wife, Catharine, and their son, Philip, were rescued at the last minute from the rapidly engulfing flames. The duke's shield bearer and the duchess's chambermaid died in the fire. Duke George of Brunswick and Lüneburg lived in the castle until 1635 with his wife, the Landgravine Anne Eleanor of Hesse-Darmstadt. In 1629, the first Elector of Hanover was born here, Ernest Augustus, Duke of Brunswick-Lüneburg. In 1714 the "schloss" was given up as a residence and from 1882 it was the seat of the district court of Herzberg. In 1900 a museum was established in the castle. Although the castle had survived largely undamaged by fire over the centuries, it suffered severe damage at the end of the Second World War, which has since been repaired. On the night of 4 April 1945, there was a mighty explosion at a nearby ammunition dump, where 40,000 kg of explosives and 8,000 mines had been stored. This blew the roof off the castle and, in the wake of the explosion, the museum was destroyed and probably looted. In 1947, further damage was caused when some nearby military bunkers were blown up. Today the castle houses a small cultural centre with a restaurant, a museum and the great hall, which is used for various cultural events. The museum portrays the history of the forestry industry in the Harz, the castle's history and that of the Welfs. Other exhibition areas present the history of Herzberg arms manufacture as well as the work of Herzberg organ maker Johann Andreas Engelhardt.