The shift change came a little before the sun actually set, though it was only swapping the two guards on the wall for a different two, with torches lit around the walls, at set intervals. It's almost as if the visible defenses are specifically trying to draw someone in, to run into the magical ones. They probably have a mage who's cunning, powerful... or both.