From the colorfully painted houses clustered around the postage stamp-sized harbor , avoid the crowds of day-trippers by setting out on a paved cliff walk .
The day-trippers typically leave at around 4 in the afternoon .