Fleece's Achilles' heel is that, without a nylon coating, it lacks any wind resistance--a slight breeze that wool and cotton would turn back cuts right through fleece with a nasty bite.
Fleece is a good material to keep you warm. 