The Palazzo Rospigliosi, in the city center, occupies the site of an ancient medieval castle, which is first referenced to when Pope Paschal II destroyed Zagarolo in the early 12th century after the rebellion of the Colonna family, of which the palace was one of the strongholds. For many centuries it remained under the Colonna influence, and was besieged and destroyed several times due to the rivalry between the papacy and the Colonna family. In the 16th century the Colonnas and the papacy started having peaceful relations, and Zagarolo became a duchy, hence the Palace became known as Palazzo Ducale.