Why GPT-3 Matters
Why does GPT-3 matter? Well, the article starts with a plot that shows what looks like an exponential growth in the number of parameters, compared with previous models. So it matters because it&#x27;s bigger.<p>Further down there&#x27;s a plot titled &quot;Aggregate Performance Across Benchmarks&quot; where we can see that performance is on average about 50%. I don&#x27;t know what the baseline for this plot should be (what is the expected average if the tasks are solved by a random classifier?) but comparing this plot with the plot at the top of the article, it doesn&#x27;t really look like there&#x27;s a huge improvement in accuracy with a huge increase in the number of parameters. In fact, it&#x27;s quite the contrary: there&#x27;s a small increase and a very smooth, almost linear curve. So that&#x27;s an exponential increase in the use of resources for an almost linear increase in performance? That&#x27;s not that impressive.<p>So it appears that the big thing about GPT-3 is that it&#x27;s big.<p>It should also be noted that the public interest about GPT-3 is mostly focused on its ability to generate text, for which there is no good metric. So basically, GPT-3 is big, but it&#x27;s not that good in tasks for which there are formal benchmarks (such as they are, because Natural Language Understanding benchmarks are often very poorly made and don&#x27;t really measure what they say they measure) and we can&#x27;t really tell how good it is in the one task that interests most people the most.