GPT-3
I don't know much about ML but I wonder what the need for so much training data (apparently about 500 billion words for GPT-3) means for this approach. Humans achieve their performance levels while only ever observing a tiny fraction of that training data --- at least in word form. I see only two possibilities:<p>-- Human brains somehow are able to learn from all forms of sensory input and world interaction in ways that pay off in word tests. (Even then, would the information processed by an average human match the GPT-3 training corpus?)<p>-- The human brain has a very different architecture that lets us learn vastly more efficiently from small amounts of training data.<p>Is there another possibility I'm missing?<p>If the latter is true then that gives humans a durable advantage over GPT-3-like approaches.