Caloric retention is pretty important when food is scarce. If you're shitting your guts out from drinking milk as an adult and losing all of the calories your body has yet to absorb you're in a really bad position. Either you are losing calories constantly from being sick, or you have to omit an entire food source from your diet which likely means you will simply be weaker and smaller than people who can drink milk.