Answers.Body: <p>This is one of the most important issues in the philosophy of artificial intelligence. </p>
  
  <p>The most famous philosophical argument that attempts to address this issue is the <a href="https://plato.stanford.edu/entries/chinese-room/" rel="noreferrer">Chinese Room argument</a> published by the philosopher John Searle in 1980. </p>
  
  <p>The argument is quite simple. Suppose that you are inside a room and you need to communicate (in a written form) with people outside the room in a certain language that you do not <em>understand</em> (in the particular example given by Searle, Chinese), but you are given the rules to manipulate the characters of this language (for a given input, you have the rules to produce the correct output). If you follow these rules, to the people outside the room, it will seem as if you understand this language, but you don't. </p>
  
  <p>To be more concrete, when I say "apple", you understand that it refers to a specific fruit because you have eaten apples and you have a model of the world. That's understanding, according to Searle.</p>
  
  <p>The most famous mathematical model of computers, the Turing machine, is essentially a system that manipulates symbols, so the Chinese Room argument directly applies to computers. </p>
  
  <p>Many replies or counterarguments to the CR argument have been discussed, such as</p>
  
  <ul>
  <li>the <a href="https://plato.stanford.edu/entries/chinese-room/#SystRepl" rel="noreferrer">system reply</a> (the symbol manipulator is only a part of the larger system).</li>
  <li>the <a href="https://plato.stanford.edu/entries/chinese-room/#RoboRepl" rel="noreferrer">robot reply</a> (the symbol manipulator does not understand the meaning of the symbols because it has not experienced the associated real-world objects, so it suggests that understanding requires a body with sensors and controllers)</li>
  <li>the <a href="https://plato.stanford.edu/entries/chinese-room/#BraiSimuRepl" rel="noreferrer">brain simulator reply</a> (the symbol manipulator can actually simulate the activity in the brain of a person that understands the unknown language)</li>
  </ul>
  
  <p>So, can we prove that machines really understand? Even before Searle, Turing had already asked the question <a href="https://academic.oup.com/mind/article/LIX/236/433/986238" rel="noreferrer">"Can machines think?"</a>. To prove this, you need a rigorous definition of understanding and thinking that people agree on. However, many people do not want to agree on a <a href="https://ai.stackexchange.com/a/13100/2444">definition of intelligence</a> and understanding (hence the many counterarguments to the CR argument). So, if you want to prove that machines understand, you need to provide a proof with respect to a specific definition of understanding. For example, if you think that understanding is just a side effect of symbol manipulation, you can easily prove that machines understand many concepts (it just follows from the definition of a Turing machine). However, even if understanding was just a side effect (what does a <em>side effect</em> actually mean in this case?) of symbol manipulation, would a machine be able to understand the same concepts and in the same way that humans understand? It's harder to answer this question because we really do not know if humans only manipulate symbols in our brains.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T11:16:59.077Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109019/discussion-on-answer-by-nbro-how-does-one-prove-comprehension-in-machines).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T02:45:10.140Z
Answers.LastActivityDate: 2020-06-03T20:39:58.710Z
Answers.LastEditDate: 2020-06-03T20:39:58.710Z
Answers.UserDisplayName: nbro
Answers.Body: <p>I recently came across a neat definition of understanding in Roger Schank's <em>Dynamic Memory</em>:</p>
  
  <p>Basically, you store everything you experience in your memory, but you need to index it in order to be able to use it for processing. Obviously, all experiences are slightly different, eg going to a restaurant is broadly the same, but the details vary. So you need to abstract away the details and store those only if necessary (eg if the food or service was particularly good or bad). Otherwise you just store a general template (or 'script') of the event.</p>
  
  <p>In your memory (note: this is modeled, not neurologically correct) you thus have a whole set of event scripts that you can retrieve. So currently I would be accessing my reply-to-stack-exchange-question script to guide me how to best write this answer without getting downvoted for ludicrous claims etc. </p>
  
  <p>Understanding, then, would be to receive (through sensory input, or language) an event, and putting it into the right area in your memory. So if I told you I just went to Burger King, you would understand it when this activates your fast-food-restaurant memory set. If I then told you I went there to wipe the floor, it should instead activate cleaning-job, rather than fast-food-restaurant. So you understand the sequence "I went to Burger King to clean the floor" by linking it to the correct memory region. If a computer then responded with "What did you eat?" it would clearly not have understood the input. But a response of "Do you get free food for working there?" would indicate some level of comprehension/understanding, as it might recognise that people working in food outlets might get free food as a work-related benefit.</p>
  
  <p>If you experience something completely new, you recognise it as a new experience, and start a new cluster of experiences. For example, if you have been to restaurants before, but never to fast food ones. The first time it will be strange and different, but you remember it as differences to the existing restaurant script. Over time it becomes strong enough (assuming to go to more fast-food restaurants), and it will become its own area, still linked to restaurants, but also not quite the same.</p>
  
  <p>What I like about this is that it is a generic mechanism, rather than an explicit processing of content. It is based on learning and experience, which I believe are key aspects of intelligent behaviour.</p>
  
  <p><strong>UPDATE:</strong> This answer is more concerned with trying to find a workable definition of what it means to <em>comprehend</em> something, rather than trying to operationalise it in a dialogue system. You can probably pass the Turing test with some clever tricks, without any comprehension at all. But the point is, what does it mean to understand something? And in the current definition it means to classify related events together, and to recognise similarities and differences between similar experiences. The reaction (ie a response) is not the understanding itself, but only a reflection of the internal state that would demonstrate understanding.</p>
  
  <p>The difference to a neural network is, I would guess, that it can cope with a broad range of experiences, where a NN would need vast amounts of training data (as it doesn't comprehend). Comprehension involves compression of information through abstraction and evaluating differences. This is still a hard problem, and I'd think difficult to achieve just with automated machine learning.</p>
  
  <p><strong>UPDATE 2:</strong> With regards to the Turing Test, in a way it goes back to deep philosophical points about empiricism. How do you know the world around you exists? You can see it. But how do you know your eyes tell you the true picture? You can quickly descend into a Matrix-like scenario where you don't know anything for certain.</p>
  
  <p>The Turing Test is a proxy for showing understanding. You don't know the computer understands what you say, so you observe its responses and interpret them accordingly. Just like at school: the teachers asks a question, and from the pupils' answers infers whether they show understanding. If you simply regurgitate a memorised answer, that's not understanding. If you paraphrase in different words, that shows some sort of comprehension. If you draw analogies to similar issues and analyse why and how they are distinct, now there you show that you really get it.</p>
  
  <p>Because we cannot inspect the internal state of a pupil, we cannot measure objectively whether they understood something. We only have communication as an interface between our mind and theirs, and so far chatbots have focused on getting that right. But I think what we really need is to work on memory and memory processing to get further towards comprehension or understanding. And I say this as a computational linguist who specialises in the language parts...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T11:17:21.300Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109020/discussion-on-answer-by-oliver-mason-how-does-one-prove-comprehension-in-machine).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T09:22:36.153Z
Answers.LastActivityDate: 2020-06-04T10:05:26.260Z
Answers.LastEditDate: 2020-06-04T10:05:26.260Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Say we have a machine and we give it a task to do (vision task, language task, game, etc.), how can one prove that a machine actually know's what's going on/happening in that specific task?</p>
  
  <p>To narrow it down, some examples:</p>
  
  <p><strong>Conversation</strong> - How would one prove that a machine actually knows what it's talking about or comprehending what is being said? The Turing test is a good start, but never actually addressed <em>actual comprehension</em>.</p>
  
  <p><strong>Vision</strong>: How could someone prove or test that a machine actually knows what it's seeing? Object detection is a start, but I'd say it's very inconclusive that a machine understands at any level what it is actually seeing.</p>
  
  <p>How do we prove comprehension in machines?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-07T11:16:50.963Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109018/discussion-on-question-by-landon-g-how-does-one-prove-comprehension-in-machines).
Comments.UserDisplayName: nbro
Tags: <philosophy><agi><artificial-consciousness><chinese-room-argument>
Title: How does one prove comprehension in machines?
UserDisplayName: Landon G
ThreadText: # How does one prove comprehension in machines?
  
  (Asked by: username_0 on 2020-06-03)
  
  Say we have a machine and we give it a task to do (vision task, language task, game, etc.), how can one prove that a machine actually know's what's going on/happening in that specific task?
  
  To narrow it down, some examples:
  
  **Conversation** - How would one prove that a machine actually knows what it's talking about or comprehending what is being said? The Turing test is a good start, but never actually addressed *actual comprehension*.
  
  **Vision**: How could someone prove or test that a machine actually knows what it's seeing? Object detection is a start, but I'd say it's very inconclusive that a machine understands at any level what it is actually seeing.
  
  How do we prove comprehension in machines?
  
  **Tags:** philosophy, agi, artificial-consciousness, chinese-room-argument
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109018/discussion-on-question-by-username_0-g-how-does-one-prove-comprehension-in-machines).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-03. Score: 15
  
  This is one of the most important issues in the philosophy of artificial intelligence.
  
  The most famous philosophical argument that attempts to address this issue is the [Chinese Room argument](https://plato.stanford.edu/entries/chinese-room/) published by the philosopher John Searle in 1980.
  
  The argument is quite simple. Suppose that you are inside a room and you need to communicate (in a written form) with people outside the room in a certain language that you do not *understand* (in the particular example given by Searle, Chinese), but you are given the rules to manipulate the characters of this language (for a given input, you have the rules to produce the correct output). If you follow these rules, to the people outside the room, it will seem as if you understand this language, but you don't.
  
  To be more concrete, when I say "apple", you understand that it refers to a specific fruit because you have eaten apples and you have a model of the world. That's understanding, according to Searle.
  
  The most famous mathematical model of computers, the Turing machine, is essentially a system that manipulates symbols, so the Chinese Room argument directly applies to computers.
  
  Many replies or counterarguments to the CR argument have been discussed, such as
  
  * the [system reply](https://plato.stanford.edu/entries/chinese-room/#SystRepl) (the symbol manipulator is only a part of the larger system).
  * the [robot reply](https://plato.stanford.edu/entries/chinese-room/#RoboRepl) (the symbol manipulator does not understand the meaning of the symbols because it has not experienced the associated real-world objects, so it suggests that understanding requires a body with sensors and controllers)
  * the [brain simulator reply](https://plato.stanford.edu/entries/chinese-room/#BraiSimuRepl) (the symbol manipulator can actually simulate the activity in the brain of a person that understands the unknown language)
  
  So, can we prove that machines really understand? Even before Searle, Turing had already asked the question ["Can machines think?"](https://academic.oup.com/mind/article/LIX/236/433/986238). To prove this, you need a rigorous definition of understanding and thinking that people agree on. However, many people do not want to agree on a [definition of intelligence](https://ai.stackexchange.com/a/13100/2444) and understanding (hence the many counterarguments to the CR argument). So, if you want to prove that machines understand, you need to provide a proof with respect to a specific definition of understanding. For example, if you think that understanding is just a side effect of symbol manipulation, you can easily prove that machines understand many concepts (it just follows from the definition of a Turing machine). However, even if understanding was just a side effect (what does a *side effect* actually mean in this case?) of symbol manipulation, would a machine be able to understand the same concepts and in the same way that humans understand? It's harder to answer this question because we really do not know if humans only manipulate symbols in our brains.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109019/discussion-on-answer-by-username_1-how-does-one-prove-comprehension-in-machines).  
  
  ---
  
  ### Answer by username_2 on 2020-06-03. Score: 5
  
  I recently came across a neat definition of understanding in Roger Schank's *Dynamic Memory*:
  
  Basically, you store everything you experience in your memory, but you need to index it in order to be able to use it for processing. Obviously, all experiences are slightly different, eg going to a restaurant is broadly the same, but the details vary. So you need to abstract away the details and store those only if necessary (eg if the food or service was particularly good or bad). Otherwise you just store a general template (or 'script') of the event.
  
  In your memory (note: this is modeled, not neurologically correct) you thus have a whole set of event scripts that you can retrieve. So currently I would be accessing my reply-to-stack-exchange-question script to guide me how to best write this answer without getting downvoted for ludicrous claims etc.
  
  Understanding, then, would be to receive (through sensory input, or language) an event, and putting it into the right area in your memory. So if I told you I just went to Burger King, you would understand it when this activates your fast-food-restaurant memory set. If I then told you I went there to wipe the floor, it should instead activate cleaning-job, rather than fast-food-restaurant. So you understand the sequence "I went to Burger King to clean the floor" by linking it to the correct memory region. If a computer then responded with "What did you eat?" it would clearly not have understood the input. But a response of "Do you get free food for working there?" would indicate some level of comprehension/understanding, as it might recognise that people working in food outlets might get free food as a work-related benefit.
  
  If you experience something completely new, you recognise it as a new experience, and start a new cluster of experiences. For example, if you have been to restaurants before, but never to fast food ones. The first time it will be strange and different, but you remember it as differences to the existing restaurant script. Over time it becomes strong enough (assuming to go to more fast-food restaurants), and it will become its own area, still linked to restaurants, but also not quite the same.
  
  What I like about this is that it is a generic mechanism, rather than an explicit processing of content. It is based on learning and experience, which I believe are key aspects of intelligent behaviour.
  
  **UPDATE:** This answer is more concerned with trying to find a workable definition of what it means to *comprehend* something, rather than trying to operationalise it in a dialogue system. You can probably pass the Turing test with some clever tricks, without any comprehension at all. But the point is, what does it mean to understand something? And in the current definition it means to classify related events together, and to recognise similarities and differences between similar experiences. The reaction (ie a response) is not the understanding itself, but only a reflection of the internal state that would demonstrate understanding.
  
  The difference to a neural network is, I would guess, that it can cope with a broad range of experiences, where a NN would need vast amounts of training data (as it doesn't comprehend). Comprehension involves compression of information through abstraction and evaluating differences. This is still a hard problem, and I'd think difficult to achieve just with automated machine learning.
  
  **UPDATE 2:** With regards to the Turing Test, in a way it goes back to deep philosophical points about empiricism. How do you know the world around you exists? You can see it. But how do you know your eyes tell you the true picture? You can quickly descend into a Matrix-like scenario where you don't know anything for certain.
  
  The Turing Test is a proxy for showing understanding. You don't know the computer understands what you say, so you observe its responses and interpret them accordingly. Just like at school: the teachers asks a question, and from the pupils' answers infers whether they show understanding. If you simply regurgitate a memorised answer, that's not understanding. If you paraphrase in different words, that shows some sort of comprehension. If you draw analogies to similar issues and analyse why and how they are distinct, now there you show that you really get it.
  
  Because we cannot inspect the internal state of a pupil, we cannot measure objectively whether they understood something. We only have communication as an interface between our mind and theirs, and so far chatbots have focused on getting that right. But I think what we really need is to work on memory and memory processing to get further towards comprehension or understanding. And I say this as a computational linguist who specialises in the language parts...
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109020/discussion-on-answer-by-username_2-username_2-how-does-one-prove-comprehension-in-machine).  
  
  ---
  
  