I think a valid point is, Disney World is pretty expensive and there's tons of people there.  You can build those core skills with your young kids at ANY populated kid friendly area.  Why waste an experience like that when your kid is a toddler and can't remember?  Just take them to parks, or museums, or like you said, a zoo.  I went on a trip to Hawaii when I was around 5 and I cant remember a thing.  I see pictures of that and get wistful cus I would love to go there now, but will not be able to afford that anywhere in my near future.

Unless you can afford to have multiple family vacations, I think many parents are better off saving their time and money until the kid is a bit older.