Analysis: ChatGPT is great at what it’s designed for. You’re just using it wrong
That depends on what you claim it is designed for.<p>Most claims about chatGPT&#x27;s behavior do not match its actual behavior.<p>Let&#x27;s consider the claim on chatGPT&#x27;s home page:<p>&gt; The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.<p>This statement is correct from the perspective of <i>utility</i>: ChatGPT does, at the end of the day, exhibit these behaviors successfully.<p>But there&#x27;s a problem here: <i>utility</i> isn&#x27;t the perspective that the average reader is likely to infer with. In fact, they are expected not to!<p>The expected perspective is <i>ability</i>. After all, we are talking about what ChatGPT is <i>able</i> to do.<p>This is the most subtle, yet most important distinction. ChatGPT is, itself, entirely <i>not</i> able to <i>do</i> any of these things!<p>But how then does it exhibit these behaviors? If it isn&#x27;t doing the work, who is, and how (and where) does ChatGPT leverage that work?<p>To answer that question, let&#x27;s talk about what our expectations are for what &quot;doing&quot; means.<p>Traditionally, to <i>do</i> a <i>thing</i>, you must identify the <i>subject</i> (the thing), and apply a <i>verb</i> (do). ChatGPT is <i>explicitly</i> capable of neither.<p>ChatGPT does not recognize subjects. ChatGPT does not implement arbitrary behaviors from the meaning of verbs.<p>Before ChatGPT can exist at all, let alone exhibit behavior, it must be trained on a dataset. Here&#x27;s what that looks like:<p>ChatGPT&#x27;s neutral net is fed a stream of text. That text is tokenized into short groups of characters. This is our first divergence from expectation: why not tokenize words and punctuation? Because <i>ChatGPT explicitly does not interact with words or grammar</i>. It does something else entirely, and short groups of characters are a good structure for that thing.<p>Next, the groups of characters (tokens) are organized as &quot;neurons&quot;. So far, it&#x27;s a graph without edges.<p>Next, duplicate nodes are found and linked together. This creates the foundational model that will later resemble (but not <i>be</i>) a knowledge graph.<p>Next, duplicate patterns of nodes and edges are recognized, and connected together. Same as the last step, just higher order. Our neutral net model is complete.<p>Finally, the neutral net is <i>trained</i>. It&#x27;s given a series of tests, each with an expected result. A successful test is weighted positively.<p>At no point did we define categories like subjects, grammar, or verbs. We didn&#x27;t even define <i>words</i>. What we did define is a model of how text relates to itself.<p>And that&#x27;s where the magic lies. What does text mean to itself? <i>Literally everything</i>. Words and grammar are the symbolic representation of ideas, and that representation is explicitly defined.<p>The only reason that isn&#x27;t enough on its own is that natural language is imperfect, and full of ambiguity. In other words, <i>natural language lies to itself about itself</i>.<p>Humans compensate with <i>logic</i> and <i>context</i>. We explicitly define the meaning of text by building up to it with backstory (context), and when we are uncertain, we filter out any construction that isn&#x27;t logical.<p>By providing a dataset of text that was logically constructed by humans, modeling how that dataset relates to itself, and explicitly training a preferred continuation whenever a human recognizes a bad continuation, what we end up with is a fuzzy algorithm that <i>implicitly</i> maps the <i>semantics</i> of an expression to <i>the most familiar semantic continuation</i>.<p>Because semantics are very nearly a 1-to-1 mapping from symbol to meaning, the <i>emergent effect</i> of this system very nearly mirrors the logic-based interaction of human writing.<p>But every ambiguity that exists in language <i>is also an emergent effect</i>. It becomes a lie or a mistake. The logical relationship that was captured by symbolic representation when a human wrote it <i>is broken</i>. Natural language has lied to itself about itself. A human could recover, but ChatGPT must be explicitly trained. This is done either by explicit training (to define a preference&#x2F;expectation), or by filtering the original dataset to be less ambiguous.<p>Language itself can also recover, by providing explicit clarification after the fact. This is implemented often in technical writing.<p>So what can ChatGPT <i>do</i>? Generate a semantically likely continuation from its training dataset.<p>What can a semantically likely continuation be? Either a logically coherent response or a semantically coherent lie.<p>How hard is it to guarantee the former? Extremely hard.