Rodney Brooks on GPT-4
This is a terrible article written by someone who doesn&#x27;t seem to have even tried GPT 4. Their only example references GPT 3.5, for example, and then they waffle on about only vaguely related topics such as level 5 self-driving.<p>This quote in particular stood out as ignorant:<p><i>“What the large language models are good at is saying what an answer should sound like, which is different from what an answer should be.”</i><p>That&#x27;s... not at all how <i>large</i> language models work. Tiny, trivial, toy language models work like this, because they don&#x27;t have the internal capacity to do anything else. They just don&#x27;t have enough parameters.<p>Stephen Wolfram explained it best: After a point, the <i>only</i> way to get better at modelling the statistics of language is to go to the level &quot;above&quot; grammar and start modelling common sense facts about the world. The larger the model, the higher the level of abstraction it can reach to improve its predictions.<p>His example was this sentence: &quot;The elephant flew to the Moon.&quot;<p>That is a syntactically and grammatically correct sentence. A toy LLM, or older NLP algorithms will mark that as &quot;valid&quot; and happily match it, predict it, or whatever. But elephants <i>don&#x27;t</i> fly to the Moon, not because the <i>sentence is invalid</i>, but because they can&#x27;t fly, the Moon has never been visited by any animal, and even humans can&#x27;t reach it (at the moment). To predict that this sentence is <i>unlikely</i>, the model has to encode all of that knowledge about the world.<p>Go ask GPT 4 -- not 3.5 -- what it thinks about elephants flying to the moon. Then, and only them go write a snarky IEEE article.