The False Promise of ChatGPT
They get very near the point, and completely miss it at the end.<p>&gt; It summarizes the standard arguments in the literature by a kind of super-autocomplete, refuses to take a stand on anything, pleads not merely ignorance but lack of intelligence<p>That is an <i>accurate</i> response from ChatGPT.<p>ChatGPT, indeed, <i>does not</i> offer intelligence. Calling it AI serves no purpose except <i>personification</i>.<p>The only thing ChatGPT <i>does</i> is &quot;some kind of super autocomplete&quot;. That&#x27;s all it <i>can do</i>.<p>The only reason ChatGPT is so impressive in the first place, is that the thing it is &quot;autocompleting&quot; - <i>language itself</i> - is impressive.<p>Language is data. Inside that data is not simply a collection of ideas: language holds the logical associations that connect ideas to each other. It can even hold an objective perspective on those associations. It can even hold an objective perspective on that objective perspective!<p>ChatGPT succeeds in finding new paths through that data, but it is walking blind: it doesn&#x27;t know <i>what</i> any of the data <i>is</i>. It only knows where.<p>It just so happens that &quot;where&quot; holds a lot of meaning. Language uses &quot;where&quot; to encode a lot of information: &quot;where&quot; is a fundamental building block for grammar itself. It&#x27;s called &quot;semantics&quot;.<p>Without any other language feature, ChatGPT is able to manipulate and apply the &quot;where&quot; of language to itself. It&#x27;s able to manipulate semantics.<p>But that&#x27;s it. It can&#x27;t do anything else. And that&#x27;s a problem. What is semantically valid might be really useful, really wrong, or really nonsensical. The only filter ChatGPT has for those categories is what it has and hasn&#x27;t read already. Most of that is not nonsense: most possible nonsense isn&#x27;t written in the first place. Most of what we write has explicit meaning. But nothing stops us from writing silly things, or even lies. Nothing stops language itself from getting the subjects of the logic mixed up. As far as language is concerned, that is not a bug: it&#x27;s a feature.