In the 17th century this fort was all there was to the town , but dwellings were gradually built up around it on reclaimed marshland .
The fort was designed to house the local noblemen and royal family .