The AI Scaling Hypothesis
There&#x27;s a deeper more troubling problem being exposed here - deep learning systems are at least an order of magnitude less data efficient than the systems they hope to replicate.<p>GPT-3 175B is trained with 499 Billion tokens[1]. Let&#x27;s assume token = word for the sake of this argument[2]. The average adult person reads at a rate of 238wmp[3]. Then a human who reads 24hrs&#x2F;day from birth until their 18th birthday would read a total of 2.2B billion words[4], or 0.45% of the words GPT-3 was trained on.<p>Human&#x27;s simply do much more with much less. So what gives? I don&#x27;t disagree that we still haven&#x27;t reached the end of what scaling can do, but there is a creeping suspicion that we&#x27;ve gotten something fundamentally wrong on the way there.<p>1. <a href="https:&#x2F;&#x2F;lambdalabs.com&#x2F;blog&#x2F;demystifying-gpt-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lambdalabs.com&#x2F;blog&#x2F;demystifying-gpt-3&#x2F;</a><p>2. GPT-based models use BPE and while we would dive into the actual dictionary of tokens and make a word-token relationship,  we both agree that although this isn&#x27;t a 1-to-1 relationship it won&#x27;t change the conclusion <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;tokenizer_summary" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;tokenizer_summary</a><p>3. <a href="https:&#x2F;&#x2F;psyarxiv.com&#x2F;xynwg&#x2F;" rel="nofollow">https:&#x2F;&#x2F;psyarxiv.com&#x2F;xynwg&#x2F;</a><p>4. 238*60*24*365*18=2,251,670,400