It was first constructed by the Bishop of Lausanne as a tower on the banks of Lake Léman around 1170. A century later, it was rebuilt and transformed into a fortified residence for bishops, particularly for Guillaume de Varax. It was also used as prison.