Your question is in two parts. First, why build a monster of a castle someplace it isn't strictly necessary, or strategically sound. Second, why would an enemy be bent on getting in when it may be easier to sit and watch you starve.
I think politics is a good answer to both. First, a castle isn't only a fortress, but a statement of intent, of power, and status. I do disagree that the garrison is more important than the brick and mortar. Suppose you are a warlord and have in your possession a very nice pile of gold. What to do with it if you aren't planning on going off to conquer more land? Invest in something very visible to tell your neighbors not to screw with you.
You build it because you can, because someone who could build a giant castle next to an unimportant but very pretty lake is someone who has power and status. The bigger the castle, the more materials used and the better maintained it is, the more others will know of your vast wealth and vigilant oversight - both being important factors when considering the strength and smarts of your opponent or rival. You make the castle strong because that's just being professional.
Alternately, the position of the castle is not strategically important, but culturally important. It is a symbol of power or tradition. Whoever holds it hold the honorary position as its steward and that grants political clout.
The second question is why would an enemy need to attack directly. That could be answered by what you have within your walls. In the game of warlord politics, power comes both in strength of arms and in legitimacy to rule. 893
You (the attacker) may have the bigger army, but that stupid castle holds the family that has ruled this land for generations. Even if you pen them in, there's always a chance they could stir something up. They could escape and raise an army, or someone may try to assassinate you to put them back in their rightful place. Maybe your next target will get important time to prepare, or a neighbor is preparing to attack while you're busy picnicking in front of a foreign castle. No, a smart warlord breaks down that castle to end their rivals just as soon as they can so they can get on to other things in peace.
Winter is coming, too. You have the advantage now, but if the snows come early, your men will starve and freeze outside of that castle. You can assault it now, end the threat and have a nice warm home for the cold months, or you can pull back to your own encampment. In the mean time, though, the winds could change. Messages could get in and out of the castle. New allies for your enemy could appear, and you may not have the advantage again.
If the attacker isn't playing the plunder and pillage game, but politics with the intent to control the land and rid themselves of their rivals, a castle holding their rivals would present a huge obstacle and threat to their ambitions, as well as being their main target. Once military and political opposition is eliminated, they can do whatever they came here to do. There may need to be a ticking clock to encourage a storming of the castle rather than a siege. Other events occurring, movement of another rival's armies, etc., or even the weather changing could make an attack imperative. If the leader believed they could lose their advantage if action isn't taken now - and they need something in that castle ASAP - a storming may be in order. 164