Try Medieval Demographics Made Easy (MDME), an excellent article written specifically as a guide to creating the population distributions one might use in Fantasy Role-Playing. Here's an example of how to use it, with your "I want one port city" benchmark, to get the rest of the numbers you might like:
I need a city with competing bookshops, for plot reasons. Two, maybe.... Okay, then this city has a population on the order of 13000 people. Let's say it's the biggest city. This makes the whole island's population somewhere around 75000 people. Let's make it densely-populated, like a Sicily: 90 ppl/mi2 means the island's size is 833 mi2. Densely populated, small island should be pretty fertile (or else why keep having so many babies!) so lets go with each mi2 settled supports 250 ppl: 300 mi2 settled for about 36% settled, 64% wilderness. No other cities, just towns numbering a half-dozen or so. Call it a 20 mi$\times$40 mi rectangle and you're basically done.
(I just used the expectation values for all rolls called for in MDME and rounded liberally. 878
You might want to tweak so as to get at your preferences. For instance, 2d4 rather than 1d8 for number of villages.)
Bonus Example, to illustrate how a different benchmark could play out: my island would take ten days to cross on foot. Okay, now we're a 240-mile diameter island, or ~45000 mi2. MDME gives me some guidance on likely densities; this is a backwater part of the empire, so I'll pick the number from medieval Britain, 40 ppl/mi2, giving me a population of 1.8 million. The Agriculture section tells me that about 10000 mi2 need to be settled in order to feed those people, so the island's about 20% settled, 80% wilderness. The Population Spread section tells me the largest city should have about 20000 people, another one around 10000 people, and there'll be about 18 towns ranging from 1000 to 8000 residents. &c. &c. &c.
Further Reading for your particular example: The Edge of the World: A Cultural History of the North Sea and the Transformation of Europe by <PERSON>, esp. pp.177--191 which give great detail of everyday life in areas dependent on fishing, on fishing vessels, and business practices related to the development of commercial fishing, &c.
Furthermore, there is a practical calculator available online here, sourcing from MDME. 222