The famed Rock Fort , the main focus of these battles , looms over the city from atop the great solid granite hill that gave the town its name .
Rock Fort lies right off the sandy coast , the first line of defense against pirates .