Answers.Body: <p>Yes it is a good analogy, as explained nicely by Neil.</p>
  <p>Regarding your second question:</p>
  <blockquote>
  <p>how far is AI from models that can actually understand (for some
  definition of &quot;understand&quot;) textual content?</p>
  </blockquote>
  <p>Here's the catch: how do we know that we (humans) are not simply very sophisticated chinese rooms?</p>
  <p>For instance suppose that current AI models are improved so much that their performance is on par to human performance, without the current catastrophic failures, yet they retain the current model architectures. Now you have an apparent paradox: they are indistinguishible from humans and yet you know that they are not &quot;understanding&quot;.</p>
  <p>Personal guess: It's chinese rooms all the way down.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T23:41:36.807Z
Answers.Text: Isn't this also what Turing was trying to demonstrate with his Imitation Game?
Answers.UserDisplayName: Barmar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T23:41:52.760Z
Answers.Text: That seems like a pretty big "if". It will probably never be possible for purely language-based models to, for example, produce a correct original proof of an unproven math theorem.
Answers.UserDisplayName: BlueRaja - Danny Pflughoeft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T00:16:02.420Z
Answers.Text: I would not bet on it being impossible, think about AlphaGo's move 37. Also, the average human is probably not able to prove a new math theorem.
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T00:32:45.697Z
Answers.Text: @Barmar Not sure about Turing intent, but my understanding is that he considered the Turing test as a definition of intelligence. If you pass it, you are intelligent. Today it is widely considered unsatisfactory, due to many techniques to fool the judges. For instance the AI may play ignorant and forgetful, which is very "human".
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T01:13:13.617Z
Answers.Text: @BlueRaja-DannyPflughoeft Coming up with a correct original proof of an unproven math theorem requires (a) creativity (read: randomness), (b) knowledge of what's correct and (c) an ability to chain things together "intelligently". I don't see which of those language-based models would be fundamentally incapable of. Prior to ChatGPT, I probably would've agreed it's extremely unlikely that a language-based model would theoretically be capable of coming up with such a proof. But now I'm not so sure any more. ChatGPT is very flawed, but it does bend and redefine what we thought the limits were.
Answers.UserDisplayName: NotThatGuy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T17:11:39.707Z
Answers.Text: Disproving anthropic general intelligence does not, alas, bring us any closer to a successful implementation of artificial general intelligence!
Answers.UserDisplayName: Tiercelet
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T18:34:23.957Z
Answers.Text: "For instance suppose that current **humans**  are improved so much that their performance is on par to **machine** performance, without the current catastrophic failures..."; for example in computations, chess, Go, memorization ...
Answers.UserDisplayName: Peter - Reinstate Monica
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T19:20:14.223Z
Answers.Text: Regarding your final personal guess, it cannot be an infinite regress like turtles all the way down since human's neural system is physical and thus limited and finite in terms of levels of such rule-based symbol manipulations which is supposedly to account for the distinct "hard qualitative" understanding of human beings.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T10:22:58.940Z
Answers.Text: @Rexcirus Note that AlphaGo and its chess equivalents are not a language-based models, but traditional tree search engine augmented by neural networks to help determine where to search. A similar approach could possibly be applied to prove theorems.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T13:32:57.093Z
Answers.Text: @Rexcirus Turing never said he ever considered passing the test to mean the machine is intelligent (whatever that means). The only purpose is to determine whether a machine can accurately imitate a human.
Answers.UserDisplayName: OrangeDog
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-30T05:20:09.153Z
Answers.Text: @user253751 I watched a video from facebook's research group on applying AI to solve math problems. The mathematician in the video mentioned how some people have tried using AI to find proofs in a very nonhuman way using what I believe is a mathematical axiomatic coding language called lean. I suppose it is some sort of pattern matching in the space of all connections between mathematical statements but the mathematician did not develop much on this aspect. There was also a paper by the Facebook research team on computing integrals using neural networks but I saw some critiques one that paper
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T11:23:07.380Z
Answers.Text: @user253751 seems like some sort of tree search but I am not sure, The paper: https://arxiv.org/abs/2205.11491 and a blog post: https://ai.facebook.com/blog/ai-math-theorem-proving/?utm_source=twitter&utm_medium=organic_social&utm_campaign=blog
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T21:30:28.197Z
Answers.LastActivityDate: 2023-02-25T21:30:28.197Z
Answers.UserDisplayName: Rexcirus
Answers.Body: <p>Searle's <a href="https://en.wikipedia.org/wiki/Chinese_room" rel="noreferrer">Chinese room</a> is <strong>not</strong> intended as a functional description of any real-world machine. Searle was a philosopher who created the Chinese room as a thought experiment to show what he considered an absurd conclusion of the <a href="https://en.wikipedia.org/wiki/Computational_theory_of_mind" rel="noreferrer">computational theory of mind</a>. The intended absurdity is that the person inside the room doesn't understand anything about the inputs or outputs, but that when just looking at the room from the outside, the room (i.e. the system person+dictionary) appears to understand Chinese.</p>
  <p>To Searle, it was clear that there was no &quot;understanding&quot; located anywhere here, and so this system was clearly not equivalent to a human consciousness that actually understands Chinese. But &quot;strong AI&quot; computationalists believe that all that matters for consciousness is the inputs and the outputs. Since Searle considers the conclusion that the room is conscious absurd, this thought experiment is supposed to be a refutation of this computationalist viewpoint of consciousness.</p>
  <p>ChatGPT and other large language models are <em>not</em> a realization of Chinese rooms. There isn't a human in there who doesn't understand English and instead uses a dictionary or set of rules to translate inputs to outputs.</p>
  <p>The point of the Chinese room is that it is clear that a) the human doesn't understand Chinese and b) the dictionary/rules are just a book that isn't conscious in itself either, otherwise it doesn't work as a reductio ad absurdum. The point of the thought experiment is that it <em>eliminates</em> anything to which we could attribute understanding - but indeed one of the replies to Searle was that it was just the <em>room itself</em> that had understanding/consciousness, and the interplay between the human and the dictionary is just analogous to the way different regions of the human brain might interact to produce the overall &quot;understanding&quot;.</p>
  <p>Instead, large language models consist of a big neural network that transforms the inputs to outputs. It's not two distinct entities like in the Chinese room - &quot;rules storage&quot;, i.e. the dictionary, and &quot;rules implementor&quot;, i.e. the human - it's one big algorithmic structure whose exact inner workings are often hard to explain for specific use cases. You may or may not assign the ability to &quot;understand&quot; to this network, but there are no identifiable substructures here as there are in Searle's room.</p>
  <p>These models, of course, raise much the same <em>questions</em> about consciousness and understanding that Searle's Chinese room does, but the room with its clear two-component structure bears no actual resemblance to how the <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="noreferrer">transformer networks</a> underlying large language models work. You might argue that these models are not conscious in the same way that Searle's room is not conscious, but your argument for <em>why</em> they aren't conscious (or why they don't &quot;understand&quot; the language they're using) needs to be very different from Searle's argument.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T15:40:06.877Z
Answers.Text: If the person can respond with perfectly reasonable sentences, they clearly do have a fairly good understanding of the relation between Chinese symbols. They just don't have an understanding of the relation between those symbols and the real world. It seems far from trivial to conclude that one of those involves understanding and the other one doesn't. If someone spends enough time in a Chinese room, and you then tell them what a few of those symbols mean (if they don't figure it out themselves), that may very well make everything fall into place to have them perfectly understand the language.
Answers.UserDisplayName: NotThatGuy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T16:27:00.933Z
Answers.Text: @NotThatGuy Note that I am careful to say that *Searle* says it is clear that there's no "understanding" here, not that this is an indisputable fact. The "Replies" section of the Wiki article I link in the first sentence has plenty of viewpoints that don't accept the Chinese room as an argument against computationalism.
Answers.UserDisplayName: ACuriousMind
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T10:23:58.820Z
Answers.Text: ChatGPT et al. are *precisely* implementations of Chinese rooms. Symbols go in, rules are applied, symbols go out, where is the understanding? The fact the rules are applied by a high-speed machine and not a human seems irrelevant.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T10:57:27.660Z
Answers.Text: @user253751 That's not a fact, that's *your judgement*: You are already certain that ChatGPT has no understanding, so you say it's just an implementation of a Chinese room. But a computationalist would say a human is *also* just a Chinese room: Words go in, a neural network (the brain) processes them, words come out. But e.g. Searle would disagree with this statement. That is, "X is an implementation of a Chinese room" is not a factual description of X, it's based in certain philosophical positions on what it means to be conscious and which things are or are not conscious.
Answers.UserDisplayName: ACuriousMind
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T12:39:36.153Z
Answers.Text: @ACuriousMind Notice that I did not say anything about humans but only about ChatGPT.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-01T02:42:45.007Z
Answers.Text: Searle did respond to one criticism by saying nothing would change if he memorized all the cards and the rules and just wrote out the responses himself from memory; that's basically your explanation of a large language model as one algorithmic structure.
Answers.UserDisplayName: John Moser
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T11:32:04.467Z
Answers.LastActivityDate: 2023-02-26T11:32:04.467Z
Answers.UserDisplayName: ACuriousMind
Answers.Body: <p>The chinese room argument is useless because it can be applied to the brain as well. Replace the slit with sensory input, the handbook with the wiring of the brain and the activity of the agent inside the room with neuron activity. In the same way the argument demonstrates that the room has no understanding it demonstrates that the brain has no understanding.</p>
  <p>My personal assessment is that LLMs like chatGPT have a true understanding of the domain they were trained on. My reasoning is that the training forced the model to squeeze all the information it can utilize to make its predictions into the limited amount of its network parameters. In this regard the incorporation of understanding is a far more efficient usage of the available space than any other kind of data compression.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T17:26:06.913Z
Answers.LastActivityDate: 2023-03-01T17:26:06.913Z
Answers.UserDisplayName: minimal
Answers.Body: <p>As many have stated, the Chinese room analogy is intended to show that any hardware + software instance that relies on rules alone (logical operators on input symbols) cannot be said to have understanding.  It is not a good analogy, and the argument does not apply well to trained neural networks. Neural networks (ChatGPT is actually two trained NNs -- an input-trained NN and output-trained NN) are produced as a result of extensive training (for ChatGPT, unsupervised training to generate a language model, and a couple of stages of supervised training on its output sentences). This training creates many billions of weights that are instantiated in the NN, and these weights are applied across the NN nodes as it 'processes' an input prompt.  From a macro view, the NN code is written in the logic of computer language, so one might conflate this with the logical operations on input signals described in the Chinese Room analogy, concluding that a NN is nothing but logical operations.  This is a mistake, in my view.  A trained NN is different in kind from the purely operational program that Searle described.  By evolving weightings through training, NNs encode an incredibly complex object that Searle could not have imagined when he created the analogy.  Does a neural network then have some kind of understanding? I think that it's clearly not anything like human understanding, but I also believe that it is at least some kind of understanding. There are many aspects to this discussion, to say the least. One main objection to Searle's argument is that if you look at individual neurons in the brain, you will not find understanding there either, but the physical brain does give rise to consciousness in some way (unless you believe in some extra 'magic' that overlays the physical brain -- something that Searle, a physicalist, would not allow).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T16:59:24.743Z
Answers.LastActivityDate: 2023-03-02T16:59:24.743Z
Answers.UserDisplayName: John Hausmann
Answers.Body: <p>I suggest it makes all the difference in the world whether 'the knowledge' was acquired by parsing many examples to get 'a feeling for what's right' (AI/learning), or by detailed instructions (algorithmic).</p>
  <p>Can you say how the exposition here relates to the Question title?</p>
  <p>What point is there but that the AI system should and the algorithmic should not be able to expand its programmed capabilities?</p>
  <p>How could anyone's understanding Chinese matter?</p>
  <p>Do you know - or know of - anyone who believes ChatGPT understands anything? More importantly, anyone who can explain where most people's understanding comes from?</p>
  <p>Do you know of people who know how to choose words based on anything but what they've seen?</p>
  <p>Can you look again at '…it mostly gives answers that are correct?</p>
  <p>Is that less or more useful/worth-while than the answers most people give?</p>
  <p>How could understanding Chinese matter, unless you were specifically asking for translations? Are you?</p>
  <p>I suggest that juggling Chinese symbols without understanding their meaning is irrelevant to ChatGPT.</p>
  <p>If what you're really Asking is how far AI might be from 'understanding' anything, why not first define 'understanding'?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T10:40:00.837Z
Answers.Text: `Do you know - or know of - anyone who believes ChatGPT understands anything?` Yes - people who don't understand how current AI works, i.e. *most of the planet*.
Answers.UserDisplayName: Ian Kemp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T00:25:24.223Z
Answers.Text: @IanKemp Ho, ho, ho. Was that pure cynicism or was there a useful point hidden in there? If you meant 'most of the planet' might believe that, had they any understanding or had ever heard of ChatGPT or anything similar, please Post some details.
Answers.UserDisplayName: Robbie Goodwin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T11:53:23.520Z
Answers.Text: @IanKemp People who know a whole lot about AI, and some about neuroscience, might also wonder about the difference between those things, and whether AI can really be said to not understand things.
Answers.UserDisplayName: NotThatGuy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T22:06:53.780Z
Answers.LastActivityDate: 2023-02-27T22:06:53.780Z
Answers.UserDisplayName: Robbie Goodwin
Answers.Body: <p>Interesting discussion, I stumbled on it reflecting on a ChatGPT experiment I did. I asked it to decrypt the word Hello, using the vignère cypher. When I asked it to provide only the answer, it simply guessed. When I let it explain all the steps it gets to the answer easily. So to link that to the chinese room, it seems that in the case of chatGPT, the man in the middle starts clueless but eventually knows a bit of chinese if there is a strong enough pattern within its answer. Maybe I'm knocking on the wrong door here but I find this result very interesting, how much more powerful could these models be if we could first get them to write out the reasoning we want from them as opposed to simply asking them to reason.</p>
  <p>TLDR: ChatGPT is similar to a chinese room but it would seem that the man in the middle has the capacity to &quot;learn chinese&quot; if provided with instructions on how to do so within the string to translate, I find that fascinating!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T17:35:19.643Z
Answers.Text: Here is the link to the full conversation: https://sharegpt.com/c/henBmRG
Answers.UserDisplayName: JP Taillon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T17:34:50.693Z
Answers.LastActivityDate: 2023-04-06T17:34:50.693Z
Answers.UserDisplayName: JP Taillon
Answers.Body: <p>Yes, the <a href="https://en.wikipedia.org/wiki/Chinese_room" rel="noreferrer">Chinese Room argument by John Searle</a> essentially demonstrates that at the very least it is hard to <em>locate</em> intelligence in a system based on its inputs and outputs. And the ChatGPT system is built very much as a machine for manipulating symbols according to opaque rules, without any grounding provided for what those symbols mean.</p>
  <p>The large language models are trained without ever getting to see, touch, or get any experience reference for any of their language components, other than yet more written language. It is much like trying to learn the meaning of a word by looking up its dictionary definition and finding that composed of other words that you don't know the meaning of, recursively without any way of resolving it. If you possessed such a dictionary and no knowledge of the words defined, you would still be able to repeat those definitions, and if they were received by someone who did understand some of the words, the result would look like reasoning and &quot;understanding&quot;. But this understanding is not yours, you are simply able to retrieve it on demand from where someone else stored it.</p>
  <p>This is also related to the <a href="https://en.wikipedia.org/wiki/Symbol_grounding_problem" rel="noreferrer">symbol grounding problem</a> in cognitive science.</p>
  <p>It is possible to argue that pragmatically the &quot;intelligence&quot; shown by the overall system is still real and resides somehow in the rules of how to manipulate the symbols. This argument and other similar ones try to side-step or dismiss some proposed hard problems in AI - for instance, by focusing on behaviour of the whole system and not trying to address the currently impossible task of asking whether any system has subjective experience. This is beyond the scope of this answer (and not really what the question is about), but it is worth noting that The Chinese Room argument has some criticism, and is not the only way to think about issues with AI systems based on language and symbols.</p>
  <p>I would agree with you that the latest language models, and ChatGPT are good example models of the The Chinese Room made real. The <em>room</em> part that is, there is no pretend human in the middle, but actually that's not hugely important - the role of the human in the Chinese room is to demonstrate that from the perspective of an entity inside the room processing a database of rules, nothing need to possess any understanding or subjective experience that is relevant to the text. Now that next-symbol predictors (which all Large Language Models are to date) are demonstrating quite sophisticated, even surprising behaviour, it may lead to some better insights into the role that symbol-to-symbol references can take in more generally intelligent systems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T13:06:02.567Z
Answers.Text: **Comments have been [moved to chat](https://chat.stackexchange.com/rooms/143244/discussion-on-answer-by-neil-slater-is-the-chinese-room-an-explanation-of-how); please do not continue the discussion here.** Before posting a comment below this one, please review the [purposes of comments](/help/privileges/comment). Comments that do not request clarification or suggest improvements usually belong as an [answer](/help/how-to-answer), on [meta], or in [chat]. Comments continuing discussion may be removed.
Answers.UserDisplayName: Mithical
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T11:24:32.197Z
Answers.LastActivityDate: 2023-02-26T21:53:55.383Z
Answers.LastEditDate: 2023-02-26T21:53:55.383Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sorry if this question makes no sense. I'm a software developer but know very little about AI.</p>
  <p>Quite a while ago, I read about the Chinese room, and the person inside who has had a lot of training/instructions how to combine symbols, and, as a result, is very good at combining symbols in a &quot;correct&quot; way, for whatever definition of correct. I said &quot;training/instructions&quot; because, for the purpose of this question, it doesn't really make a difference if the &quot;knowledge&quot; was acquired by parsing many many examples and getting a &quot;feeling&quot; for what's right and what's wrong (AI/learning), or by a very detailed set of instructions (algorithmic).</p>
  <p>So, the person responds with perfectly reasonable sentences, without ever understanding Chinese, or the content of its input.</p>
  <p>Now, as far as I understand ChatGPT (and I might be completely wrong here), that's exactly what ChatGPT does. It has been trained on a huge corpus of text, and thus has a very good feeling which words go together well and which don't, and, given a sentence, what's the most likely continuation of this sentence. But that doesn't really mean it understands the content of the sentence, it only knows how to chose words based on what it has seen. And because it doesn't really understand any content, it mostly gives answers that are correct, but sometimes it's completely off because it &quot;doesn't really understand Chinese&quot; and doesn't know what it's talking about.</p>
  <p>So, my question: is this &quot;juggling of Chinese symbols without understanding their meaning&quot; an adequate explanation of how ChatGPT works, and if not, where's the difference? And if yes, how far is AI from models that can actually understand (for some definition of &quot;understand&quot;) textual content?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-25T22:03:20.810Z
Comments.Text: You are already anthropomorphizing when you asay it has a "good feeling"!
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-27T08:45:04.213Z
Comments.Text: Considering how poor ChatGPT is at counting (e.g. stating "Here are 5 examples why:" and listing anywhere between 3-7 examples) and graphical AI in general at drawing hands (counting fingers is hard), there's definitely some degree of "without understanding their meaning" indeed.
Comments.UserDisplayName: Mast
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-27T18:37:03.693Z
Comments.Text: I'm surprised that most discussions of the Chinese Room thought experiment do not mention the Turing test. If it barks like a dog, smells like a dog and licks my hand like a dog, it's a dog. The beauty of that is that we don't need to get into the nitty gritty details of what a dog "really" is. If it's silicon based, so be it. Saying "but I opened it, it's not a *real* dog!" entirely misses Turing's elegant point.
Comments.UserDisplayName: Peter - Reinstate Monica
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-28T13:37:47.210Z
Comments.Text: We don’t have a concrete model of intelligence. We don’t have a mathematical theory of intelligence. All we have are some thought experiments and ill-defined philosophical concepts. Historically, when humans try to build models of the real world based on their amazing deductive powers without evidence, they come up with stuff like the world being made of someone’s armpit or, at best, Lamarckian evolution.
Comments.UserDisplayName: GregRos
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-01T21:32:35.773Z
Comments.Text: By the way, 'combining symbols in a correct way' describes what, but learning the language?
  
  Did you follow the Turing Test?
  
  How could you show that one who responds with 'perfectly reasonable sentences' has no understanding of Chinese, or - whatever that might mean in Chinese or English - 'the content of its input'?
  
  I suggest it's not for the Question, but for any Answer that it doesn't matter whether the knowledge was acquired by parsing many examples to get a 'feeling' for what's right (AI/learning), or by a very detailed set of instructions (algorithmic).
Comments.UserDisplayName: Robbie Goodwin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-27T16:01:20.713Z
Comments.Text: The purpose and function of intelligence is to solve problems. There are various mental capacities which enable one to do so more or less effectively, which are measured when attempting to measure intelligence. People are conflating intelligence and self-awareness.
Comments.UserDisplayName: Tanath
Tags: <agi><philosophy><natural-language-understanding><chatgpt><chinese-room-argument>
Title: Is the "Chinese room" an explanation of how ChatGPT works?
UserDisplayName: Guntram Blohm
ThreadText: # Is the "Chinese room" an explanation of how ChatGPT works?
  
  (Asked by: username_0 on 2023-02-25)
  
  Sorry if this question makes no sense. I'm a software developer but know very little about AI.
  
  Quite a while ago, I read about the Chinese room, and the person inside who has had a lot of training/instructions how to combine symbols, and, as a result, is very good at combining symbols in a "correct" way, for whatever definition of correct. I said "training/instructions" because, for the purpose of this question, it doesn't really make a difference if the "knowledge" was acquired by parsing many many examples and getting a "feeling" for what's right and what's wrong (AI/learning), or by a very detailed set of instructions (algorithmic).
  
  So, the person responds with perfectly reasonable sentences, without ever understanding Chinese, or the content of its input.
  
  Now, as far as I understand ChatGPT (and I might be completely wrong here), that's exactly what ChatGPT does. It has been trained on a huge corpus of text, and thus has a very good feeling which words go together well and which don't, and, given a sentence, what's the most likely continuation of this sentence. But that doesn't really mean it understands the content of the sentence, it only knows how to chose words based on what it has seen. And because it doesn't really understand any content, it mostly gives answers that are correct, but sometimes it's completely off because it "doesn't really understand Chinese" and doesn't know what it's talking about.
  
  So, my question: is this "juggling of Chinese symbols without understanding their meaning" an adequate explanation of how ChatGPT works, and if not, where's the difference? And if yes, how far is AI from models that can actually understand (for some definition of "understand") textual content?
  
  **Tags:** agi, philosophy, natural-language-understanding, chatgpt, chinese-room-argument
  
  > **Comments:**
  > **username_1**: You are already anthropomorphizing when you asay it has a "good feeling"!  
  > **username_2**: Considering how poor ChatGPT is at counting (e.g. stating "Here are 5 examples why:" and listing anywhere between 3-7 examples) and graphical AI in general at drawing hands (counting fingers is hard), there's definitely some degree of "without understanding their meaning" indeed.  
  > **username_3**: I'm surprised that most discussions of the Chinese Room thought experiment do not mention the Turing test. If it barks like a dog, smells like a dog and licks my hand like a dog, it's a dog. The beauty of that is that we don't need to get into the nitty gritty details of what a dog "really" is. If it's silicon based, so be it. Saying "but I opened it, it's not a *real* dog!" entirely misses Turing's elegant point.  
  > **username_4**: We don’t have a concrete model of intelligence. We don’t have a mathematical theory of intelligence. All we have are some thought experiments and ill-defined philosophical concepts. Historically, when humans try to build models of the real world based on their amazing deductive powers without evidence, they come up with stuff like the world being made of someone’s armpit or, at best, Lamarckian evolution.  
  > **username_5**: By the way, 'combining symbols in a correct way' describes what, but learning the language?  Did you follow the Turing Test?  How could you show that one who responds with 'perfectly reasonable sentences' has no understanding of Chinese, or - whatever that might mean in Chinese or English - 'the content of its input'?  I suggest it's not for the Question, but for any Answer that it doesn't matter whether the knowledge was acquired by parsing many examples to get a 'feeling' for what's right (AI/learning), or by a very detailed set of instructions (algorithmic).  
  > **username_6**: The purpose and function of intelligence is to solve problems. There are various mental capacities which enable one to do so more or less effectively, which are measured when attempting to measure intelligence. People are conflating intelligence and self-awareness.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_21 on 2023-02-25. Score: 37 | Accepted answer
  
  Yes, the [Chinese Room argument by username_16 Searle](https://en.wikipedia.org/wiki/Chinese_room) essentially demonstrates that at the very least it is hard to *locate* intelligence in a system based on its inputs and outputs. And the ChatGPT system is built very much as a machine for manipulating symbols according to opaque rules, without any grounding provided for what those symbols mean.
  
  The large language models are trained without ever getting to see, touch, or get any experience reference for any of their language components, other than yet more written language. It is much like trying to learn the meaning of a word by looking up its dictionary definition and finding that composed of other words that you don't know the meaning of, recursively without any way of resolving it. If you possessed such a dictionary and no knowledge of the words defined, you would still be able to repeat those definitions, and if they were received by someone who did understand some of the words, the result would look like reasoning and "understanding". But this understanding is not yours, you are simply able to retrieve it on demand from where someone else stored it.
  
  This is also related to the [symbol grounding problem](https://en.wikipedia.org/wiki/Symbol_grounding_problem) in cognitive science.
  
  It is possible to argue that pragmatically the "intelligence" shown by the overall system is still real and resides somehow in the rules of how to manipulate the symbols. This argument and other similar ones try to side-step or dismiss some proposed hard problems in AI - for instance, by focusing on behaviour of the whole system and not trying to address the currently impossible task of asking whether any system has subjective experience. This is beyond the scope of this answer (and not really what the question is about), but it is worth noting that The Chinese Room argument has some criticism, and is not the only way to think about issues with AI systems based on language and symbols.
  
  I would agree with you that the latest language models, and ChatGPT are good example models of the The Chinese Room made real. The *room* part that is, there is no pretend human in the middle, but actually that's not hugely important - the role of the human in the Chinese room is to demonstrate that from the perspective of an entity inside the room processing a database of rules, nothing need to possess any understanding or subjective experience that is relevant to the text. Now that next-symbol predictors (which all Large Language Models are to date) are demonstrating quite sophisticated, even surprising behaviour, it may lead to some better insights into the role that symbol-to-symbol references can take in more generally intelligent systems.
  
  > **Comments:**
  > **username_22**: **Comments have been [moved to chat](https://chat.stackexchange.com/rooms/143244/discussion-on-answer-by-username_21-is-the-chinese-room-an-explanation-of-how); please do not continue the discussion here.** Before posting a comment below this one, please review the [purposes of comments](/help/privileges/comment). Comments that do not request clarification or suggest improvements usually belong as an [answer](/help/how-to-answer), on [meta], or in [chat]. Comments continuing discussion may be removed.  
  
  ---
  
  ### Answer by username_7 on 2023-02-25. Score: 21
  
  Yes it is a good analogy, as explained nicely by username_21.
  
  Regarding your second question:
  
  > how far is AI from models that can actually understand (for some
  > definition of "understand") textual content?
  
  Here's the catch: how do we know that we (humans) are not simply very sophisticated chinese rooms?
  
  For instance suppose that current AI models are improved so much that their performance is on par to human performance, without the current catastrophic failures, yet they retain the current model architectures. Now you have an apparent paradox: they are indistinguishible from humans and yet you know that they are not "understanding".
  
  Personal guess: It's chinese rooms all the way down.
  
  > **Comments:**
  > **username_8**: Isn't this also what Turing was trying to demonstrate with his Imitation Game?  
  > **username_9**: That seems like a pretty big "if". It will probably never be possible for purely language-based models to, for example, produce a correct original proof of an unproven math theorem.  
  > **username_7**: I would not bet on it being impossible, think about AlphaGo's move 37. Also, the average human is probably not able to prove a new math theorem.  
  > **username_7**: @username_8 Not sure about Turing intent, but my understanding is that he considered the Turing test as a definition of intelligence. If you pass it, you are intelligent. Today it is widely considered unsatisfactory, due to many techniques to fool the judges. For instance the AI may play ignorant and forgetful, which is very "human".  
  > **username_10**: @username_9 Coming up with a correct original proof of an unproven math theorem requires (a) creativity (read: randomness), (b) knowledge of what's correct and (c) an ability to chain things together "intelligently". I don't see which of those language-based models would be fundamentally incapable of. Prior to ChatGPT, I probably would've agreed it's extremely unlikely that a language-based model would theoretically be capable of coming up with such a proof. But now I'm not so sure any more. ChatGPT is very flawed, but it does bend and redefine what we thought the limits were.  
  > **username_11**: Disproving anthropic general intelligence does not, alas, bring us any closer to a successful implementation of artificial general intelligence!  
  > **username_3**: "For instance suppose that current **humans**  are improved so much that their performance is on par to **machine** performance, without the current catastrophic failures..."; for example in computations, chess, Go, memorization ...  
  > **username_12**: Regarding your final personal guess, it cannot be an infinite regress like turtles all the way down since human's neural system is physical and thus limited and finite in terms of levels of such rule-based symbol manipulations which is supposedly to account for the distinct "hard qualitative" understanding of human beings.  
  > **username_1**: @username_7 Note that AlphaGo and its chess equivalents are not a language-based models, but traditional tree search engine augmented by neural networks to help determine where to search. A similar approach could possibly be applied to prove theorems.  
  > **username_13**: @username_7 Turing never said he ever considered passing the test to mean the machine is intelligent (whatever that means). The only purpose is to determine whether a machine can accurately imitate a human.  
  > **username_14**: @user253751 I watched a video from facebook's research group on applying AI to solve math problems. The mathematician in the video mentioned how some people have tried using AI to find proofs in a very nonhuman way using what I believe is a mathematical axiomatic coding language called lean. I suppose it is some sort of pattern matching in the space of all connections between mathematical statements but the mathematician did not develop much on this aspect. There was also a paper by the Facebook research team on computing integrals using neural networks but I saw some critiques one that paper  
  > **username_14**: @user253751 seems like some sort of tree search but I am not sure, The paper: https://arxiv.org/abs/2205.11491 and a blog post: https://ai.facebook.com/blog/ai-math-theorem-proving/?utm_source=twitter&utm_medium=organic_social&utm_campaign=blog  
  
  ---
  
  ### Answer by username_15 on 2023-02-26. Score: 14
  
  Searle's [Chinese room](https://en.wikipedia.org/wiki/Chinese_room) is **not** intended as a functional description of any real-world machine. Searle was a philosopher who created the Chinese room as a thought experiment to show what he considered an absurd conclusion of the [computational theory of mind](https://en.wikipedia.org/wiki/Computational_theory_of_mind). The intended absurdity is that the person inside the room doesn't understand anything about the inputs or outputs, but that when just looking at the room from the outside, the room (i.e. the system person+dictionary) appears to understand Chinese.
  
  To Searle, it was clear that there was no "understanding" located anywhere here, and so this system was clearly not equivalent to a human consciousness that actually understands Chinese. But "strong AI" computationalists believe that all that matters for consciousness is the inputs and the outputs. Since Searle considers the conclusion that the room is conscious absurd, this thought experiment is supposed to be a refutation of this computationalist viewpoint of consciousness.
  
  ChatGPT and other large language models are *not* a realization of Chinese rooms. There isn't a human in there who doesn't understand English and instead uses a dictionary or set of rules to translate inputs to outputs.
  
  The point of the Chinese room is that it is clear that a) the human doesn't understand Chinese and b) the dictionary/rules are just a book that isn't conscious in itself either, otherwise it doesn't work as a reductio ad absurdum. The point of the thought experiment is that it *eliminates* anything to which we could attribute understanding - but indeed one of the replies to Searle was that it was just the *room itself* that had understanding/consciousness, and the interplay between the human and the dictionary is just analogous to the way different regions of the human brain might interact to produce the overall "understanding".
  
  Instead, large language models consist of a big neural network that transforms the inputs to outputs. It's not two distinct entities like in the Chinese room - "rules storage", i.e. the dictionary, and "rules implementor", i.e. the human - it's one big algorithmic structure whose exact inner workings are often hard to explain for specific use cases. You may or may not assign the ability to "understand" to this network, but there are no identifiable substructures here as there are in Searle's room.
  
  These models, of course, raise much the same *questions* about consciousness and understanding that Searle's Chinese room does, but the room with its clear two-component structure bears no actual resemblance to how the [transformer networks](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) underlying large language models work. You might argue that these models are not conscious in the same way that Searle's room is not conscious, but your argument for *why* they aren't conscious (or why they don't "understand" the language they're using) needs to be very different from Searle's argument.
  
  > **Comments:**
  > **username_10**: If the person can respond with perfectly reasonable sentences, they clearly do have a fairly good understanding of the relation between Chinese symbols. They just don't have an understanding of the relation between those symbols and the real world. It seems far from trivial to conclude that one of those involves understanding and the other one doesn't. If someone spends enough time in a Chinese room, and you then tell them what a few of those symbols mean (if they don't figure it out themselves), that may very well make everything fall into place to have them perfectly understand the language.  
  > **username_15**: @username_10 Note that I am careful to say that *Searle* says it is clear that there's no "understanding" here, not that this is an indisputable fact. The "Replies" section of the Wiki article I link in the first sentence has plenty of viewpoints that don't accept the Chinese room as an argument against computationalism.  
  > **username_1**: ChatGPT et al. are *precisely* implementations of Chinese rooms. Symbols go in, rules are applied, symbols go out, where is the understanding? The fact the rules are applied by a high-speed machine and not a human seems irrelevant.  
  > **username_15**: @user253751 That's not a fact, that's *your judgement*: You are already certain that ChatGPT has no understanding, so you say it's just an implementation of a Chinese room. But a computationalist would say a human is *also* just a Chinese room: Words go in, a neural network (the brain) processes them, words come out. But e.g. Searle would disagree with this statement. That is, "X is an implementation of a Chinese room" is not a factual description of X, it's based in certain philosophical positions on what it means to be conscious and which things are or are not conscious.  
  > **username_1**: @username_15 Notice that I did not say anything about humans but only about ChatGPT.  
  > **username_16**: Searle did respond to one criticism by saying nothing would change if he memorized all the cards and the rules and just wrote out the responses himself from memory; that's basically your explanation of a large language model as one algorithmic structure.  
  
  ---
  
  ### Answer by username_17 on 2023-03-01. Score: 7
  
  The chinese room argument is useless because it can be applied to the brain as well. Replace the slit with sensory input, the handbook with the wiring of the brain and the activity of the agent inside the room with neuron activity. In the same way the argument demonstrates that the room has no understanding it demonstrates that the brain has no understanding.
  
  My personal assessment is that LLMs like chatGPT have a true understanding of the domain they were trained on. My reasoning is that the training forced the model to squeeze all the information it can utilize to make its predictions into the limited amount of its network parameters. In this regard the incorporation of understanding is a far more efficient usage of the available space than any other kind of data compression.
  
  
  ---
  
  ### Answer by username_18 on 2023-03-02. Score: 2
  
  As many have stated, the Chinese room analogy is intended to show that any hardware + software instance that relies on rules alone (logical operators on input symbols) cannot be said to have understanding. It is not a good analogy, and the argument does not apply well to trained neural networks. Neural networks (ChatGPT is actually two trained NNs -- an input-trained NN and output-trained NN) are produced as a result of extensive training (for ChatGPT, unsupervised training to generate a language model, and a couple of stages of supervised training on its output sentences). This training creates many billions of weights that are instantiated in the NN, and these weights are applied across the NN nodes as it 'processes' an input prompt. From a macro view, the NN code is written in the logic of computer language, so one might conflate this with the logical operations on input signals described in the Chinese Room analogy, concluding that a NN is nothing but logical operations. This is a mistake, in my view. A trained NN is different in kind from the purely operational program that Searle described. By evolving weightings through training, NNs encode an incredibly complex object that Searle could not have imagined when he created the analogy. Does a neural network then have some kind of understanding? I think that it's clearly not anything like human understanding, but I also believe that it is at least some kind of understanding. There are many aspects to this discussion, to say the least. One main objection to Searle's argument is that if you look at individual neurons in the brain, you will not find understanding there either, but the physical brain does give rise to consciousness in some way (unless you believe in some extra 'magic' that overlays the physical brain -- something that Searle, a physicalist, would not allow).
  
  
  ---
  
  ### Answer by username_5 on 2023-02-27. Score: 1
  
  I suggest it makes all the difference in the world whether 'the knowledge' was acquired by parsing many examples to get 'a feeling for what's right' (AI/learning), or by detailed instructions (algorithmic).
  
  Can you say how the exposition here relates to the Question title?
  
  What point is there but that the AI system should and the algorithmic should not be able to expand its programmed capabilities?
  
  How could anyone's understanding Chinese matter?
  
  Do you know - or know of - anyone who believes ChatGPT understands anything? More importantly, anyone who can explain where most people's understanding comes from?
  
  Do you know of people who know how to choose words based on anything but what they've seen?
  
  Can you look again at '…it mostly gives answers that are correct?
  
  Is that less or more useful/worth-while than the answers most people give?
  
  How could understanding Chinese matter, unless you were specifically asking for translations? Are you?
  
  I suggest that juggling Chinese symbols without understanding their meaning is irrelevant to ChatGPT.
  
  If what you're really Asking is how far AI might be from 'understanding' anything, why not first define 'understanding'?
  
  > **Comments:**
  > **username_19**: `Do you know - or know of - anyone who believes ChatGPT understands anything?` Yes - people who don't understand how current AI works, i.e. *most of the planet*.  
  > **username_5**: @username_19 Ho, ho, ho. Was that pure cynicism or was there a useful point hidden in there? If you meant 'most of the planet' might believe that, had they any understanding or had ever heard of ChatGPT or anything similar, please Post some details.  
  > **username_10**: @username_19 People who know a whole lot about AI, and some about neuroscience, might also wonder about the difference between those things, and whether AI can really be said to not understand things.  
  
  ---
  
  ### Answer by username_20 on 2023-04-06. Score: -2
  
  Interesting discussion, I stumbled on it reflecting on a ChatGPT experiment I did. I asked it to decrypt the word Hello, using the vignère cypher. When I asked it to provide only the answer, it simply guessed. When I let it explain all the steps it gets to the answer easily. So to link that to the chinese room, it seems that in the case of chatGPT, the man in the middle starts clueless but eventually knows a bit of chinese if there is a strong enough pattern within its answer. Maybe I'm knocking on the wrong door here but I find this result very interesting, how much more powerful could these models be if we could first get them to write out the reasoning we want from them as opposed to simply asking them to reason.
  
  TLDR: ChatGPT is similar to a chinese room but it would seem that the man in the middle has the capacity to "learn chinese" if provided with instructions on how to do so within the string to translate, I find that fascinating!
  
  > **Comments:**
  > **username_20**: Here is the link to the full conversation: https://sharegpt.com/c/henBmRG  
  
  ---
  
  