From the end of the 11th century, the Viscounts of Labourd begin building a castle around the three Roman towers and walls around the town, as their residence. The castle became English upon the Duchy of Aquitaine, being inherited by the English crown. After the French took occupation of the castle and town in 1294, an English force led by John St John, laid siege to the town.