The town of Edinburgh eventually spread out below the castle , with a main street leading out the entrance and down to Holyrood Palace .
It was an easy trip between the castle and the town , often taken by the villagers daily .