Aenwyn, after the scouting and the hoos back, you get the idea that humanoid figures patrol along the top of the walls. You don't see any at the moment, but they could be walkin in at any time. You do notice, too, on more careful observation, that the area between the forest and the wall isn't well maintained. There's plenty of bushes and low grass that someone clever could use to get closer. However, from above, Brom also can see something that isn't obvious where you are. It's somewhat disguised, but there's a moat surrounding the wall, too, except for the area in front of the gate.