Are we in an AI Overhang?
It think it is all right except for the A.I. part.<p>GPT-3 is taking a graph-structured object (&quot;language&quot; inclusive of syntax and semantics) over a variable-length discrete domain and crushing it into a high-dimensional vector in a continuous euclidean space.  That&#x27;s like fitting the 3-d spherical earth onto a 2-d map;  any way you do it you do violence to the map.<p>I think systems like GPT-3 are approaching an asymptote.  You could put 10x the resources in and get 10% better results,  another 10x and get 1% better results,  something like that.<p>You might do better with multi-task learning oriented towards specific useful functions (e.g. &quot;is this period the end of a sentence?&quot;) but the training problem for GPT-3 is by no means sufficient for text understanding.<p>GPT-3 fascinates people for various reasons,  one of them being almost good enough at language,  lacking understanding,  faking it,  and being the butt of a joke.<p>If GPT-3 were a person with similar language skills and people blogged about that person,  mocking it&#x27;s output,  the way we do with GPT-3,  people would find that cringeworthy.  Neurotypicals welcome it as one of their own,  and aspies envy it because it can pass better than they can.<p>At $2 a page it can replace richmansplainers such as Graham and Thiel who never listen.  It&#x27;s not a solution for folks like like Phillip Greenspun who read the comments on their blogs.<p>For that matter,  it may very well model the mindlessness of corporate America:  if you accept GPT-3 you prove you will see the Emperor&#x27;s clothes no matter how buck naked he is.  AT&amp;T executives had a perfectly good mobile phone business:  what possessed them to buy a failing satellite TV business?  Could GPT-3 replace that &quot;thinking&quot; at $2 a page?  Such a bargain.