The town of Edinburgh eventually spread out below the castle , with a main street leading out the entrance and down to Holyrood Palace .
The castle loomed over nothing as no town was connected to it anymore .