GPT-4 can't reason
Lot&#x27;s of people in this thread seem desperate to not think clearly about this.<p>Whether these tests, verbatim produce the same response on any given version isn&#x27;t the point. GPT4 doesn&#x27;t engage in reasoning if it gets any answers right. Being &quot;right&quot; isn&#x27;t a sign of reasoning. Given a dictionary mapping from questions to answers the index operation gets answers right, but it isn&#x27;t reasoning.<p>The purpose of the paper is to exhaustively list <i>proxy</i> indicators of reasoning. <i>Clearly</i> other tests will fail in every class listed, because the LLM isn&#x27;t engaged in reasoning. Since LLMs are stocahstic you shouldnt expect &quot;reporduction&quot; in the same sense. The paper provides classes of problems.<p>To reproduce it you only need to find a minor permutation of the problem in each class. But people subject to gross confirmation-bias seem only to seek out prompts which produce the right answers.<p>P(next|prev), ie., P(answer word | prompt words) is just a dictionary lookup, and that&#x27;s the optimisation objective for an LLM.<p>It turns out sequences of inferences eg., {Socrates is a Man, All men are mortal, thef. Socrates is mortal} can be modelled by dictionary lookups of the above form -- but we&#x27;re not interested in whether the answer can be found from the prompt. Whether this sequence of props can be stored as a sequence of index operations.<p>We&#x27;re interested in whether the system reasons. ie., whether the sequencing operation is inference by logical necessity.<p>It&#x27;s incomprehensible to me how gullible people are around AI today -- the Eliza effect coupled with a pseudoscientific impulse to &quot;whatever appears to work&quot;.