Answers.Body: <p>Firstly, an AGI could conceivably exhibit all of the observable properties of intelligence without being conscious. Although that may seem counter-intuitive, at present we have no physical theory that allows us to detect consciousness (philosophically speaking, a <a href="http://plato.stanford.edu/entries/zombies/" rel="nofollow noreferrer">'Zombie'</a> is indistinguishable from a non-Zombie - see the writing of <a href="http://rads.stackoverflow.com/amzn/click/0316180661" rel="nofollow noreferrer">Daniel Dennett</a> and <a href="http://consc.net/books/tcm/" rel="nofollow noreferrer">David Chalmers</a> for more on this). Destroying a non-conscious entity has the same moral cost as destroying a chair.</p>
  
  <p>Also, note that 'destroy' doesn't necessarily mean the same for entities with <em>persistent substrate</em>, i.e. their 'brain state' can be reversibly serialized to some other storage medium and/or multiple copies of them can co-exist. So if by 'destroy' we simply mean 'switch off', then an AGI might conceivably be reassured of a subsequent re-awakening. Douglas Hofstadter gives an interesting description of such an 'episodic consciousness' in <a href="http://themindi.blogspot.co.uk/2007/02/chapter-26-conversation-with-einsteins.html" rel="nofollow noreferrer">"A Conversation with Einstein's Brain"</a></p>
  
  <p>If by 'destroy', we mean 'irrevocably erase with no chance of re-awakening', then (unless we have a physical test which proves it is <em>not</em> conscious) destroying an entity with a seemingly human-level awareness is clearly morally tantamount to murder. To believe otherwise would be <em>substrate-ist</em> - a moral stance which may one day be seen as antiquated as racism.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-02T22:34:32.223Z
Answers.Text: Great answer, although I will posit that the act of destruction, even with an non-conscious entity, is not necessarily neutral from a subjective perspective.  I thinking about ideas like "intention" in a Buddhist context.  I find that, as I get older and, ostensibly, become more "evolved", I have become more and more reluctant to rack up civilian casualties in games like Grand Theft Auto. Is this is merely a form of anthropomorphism, or is there some underlying, moral imperative?
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T12:16:40.717Z
Answers.Text: pzombies are kind of a debated thing though; it's rather weird to say that you can know that an AGI has no consciousness if, following Chalmers words, there is no difference in the actions taken by zombies and people; if it is unethical to kill other people, but you cannot verify if they are pzombie or person, how is it ethical to kill an AGI? It seems you come to the same conclusion in your answer though, after rereading, so I'm gonna upvote.
Answers.UserDisplayName: Pimgd
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T15:10:54.013Z
Answers.Text: @Pimgd - We're actually in agreement (like Dennett, I happen to think that Zombies are logically impossible). If an AGI appears indistinguishable from a human, it would be unethical to kill it unless you can prove it has no consciousness. No-one knows what such a proof might look like, or even if one exists.
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T08:06:28.653Z
Answers.LastActivityDate: 2016-12-03T07:46:15.380Z
Answers.LastEditDate: 2016-12-03T07:46:15.380Z
Answers.UserDisplayName: NietzscheanAI
Answers.Body: <p>Even if machines with true Artificial General Intelligence were created, their apparent intelligence would still be by definition <em>artificial</em>. The word <em>simulation</em> is a <a href="http://www.thesaurus.com/browse/artificial" rel="nofollow noreferrer">synonym</a> and could be used to redefine AGI as Simulated General Intelligence. </p>
  
  <p>Keeping that in mind, <strong>a machine that appears to be expressing emotions would only be the result of a series of complicated algorithms</strong> allowing a computer to assess the situation and respond in an intellectually appropriate manner based on external stimulus and conditions. Every possible action this machine could possibly make would be derived from a list of possible actions the machine is capable of, no matter how large the number of possible actions grows. The machine is still a series sensors, programmed instructions, and cycles of execution. </p>
  
  <p>Destroying such a machine could potentially be the destruction of property if it wasn't owned by the person who destroyed it, but would it be murder? No. </p>
  
  <p>A broken machine can potentially be rebuilt and reactivated if it is broken. It never really died; it was destroyed. A living being that is killed is really <em>dead</em> and cannot be rebuilt and made alive once again. These key differences lead me to agree with the previous answer and conclude that <strong>no, destroying an artificial intelligence without its consent would not be murder</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-02T13:18:44.903Z
Answers.LastActivityDate: 2017-03-11T00:14:25.140Z
Answers.LastEditDate: 2017-03-11T00:14:25.140Z
Answers.UserDisplayName: Christian Westbrook
Answers.Body: <p>Yes we can, just like we sentence living human to death. Moral or not depends on the particular time point we are in history.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T07:41:10.560Z
Answers.LastActivityDate: 2023-04-12T07:41:10.560Z
Answers.UserDisplayName: lpounng
Answers.Body: <p>There is another problem besides consciousness, which is personal identity.</p>
  <p>Consider an AI whose task is learning the discourse of a particular person during years, so as to simulate their conversation as closely as possible. After the death of the seed human, their artificial continuation can keep chatting with other humans (especially family members) using the ideas taught by the now dead person.</p>
  <p>There is a strong link between such an intelligence and a human being, so it poses a problem harder than deleting a general chatbot with no individual attachment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-22T21:56:21.837Z
Answers.LastActivityDate: 2023-04-12T06:22:24.093Z
Answers.LastEditDate: 2023-04-12T06:22:24.093Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>According to Wikipedia <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence" rel="nofollow noreferrer">Artificial general intelligence(AGI)</a></p>
  
  <blockquote>
    <p>Artificial general intelligence (AGI) is the intelligence of a
    (hypothetical) machine that could successfully perform any
    intellectual task that a human being can. </p>
  </blockquote>
  
  <p>According to the below image (a screenshot of a picture from <a href="http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=251030664B23757C5D420F9ED97AB725?doi=10.1.1.136.7883&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">When will computer hardware match the human brain?</a> (1998) by Hans Moravec, and Kurzweill also uses this diagram in his book <a href="https://en.wikipedia.org/wiki/The_Singularity_Is_Near" rel="nofollow noreferrer">The Singularity Is Near: When Humans Transcend Biology</a>), today's artificial intelligence is the same as that of lizards.</p>
  
  <p><a href="https://i.sstatic.net/gddKB.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gddKB.jpg" alt="enter image description here"></a></p>
  
  <p>Let's assume that within 10-20 years, we, humans, are successful in creating an AGI, that is, an AI with human-level intelligence and emotions.</p>
  
  <p>At that point, could we destroy an AGI without its consent? Would this be considered murder?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-10T01:10:37.403Z
Comments.Text: the picture is misleading: having large amounts of compute is not equivalent to being "intelligent"
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Tags: <philosophy><agi><ethics>
Title: Can we destroy an artificial general intelligence without its consent?
UserDisplayName: Eka
ThreadText: # Can we destroy an artificial general intelligence without its consent?
  
  (Asked by: username_0 on 2016-08-04)
  
  According to Wikipedia [Artificial general intelligence(AGI)](https://en.wikipedia.org/wiki/Artificial_general_intelligence)
  
  > Artificial general intelligence (AGI) is the intelligence of a
  > (hypothetical) machine that could successfully perform any
  > intellectual task that a human being can.
  
  According to the below image (a screenshot of a picture from [When will computer hardware match the human brain?](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=251030664B23757C5D420F9ED97AB725?doi=10.1.1.136.7883&rep=rep1&type=pdf) (1998) by Hans Moravec, and Kurzweill also uses this diagram in his book [The Singularity Is Near: When Humans Transcend Biology](https://en.wikipedia.org/wiki/The_Singularity_Is_Near)), today's artificial intelligence is the same as that of lizards.
  
  [![enter image description here](https://i.sstatic.net/gddKB.jpg)](https://i.sstatic.net/gddKB.jpg)
  
  Let's assume that within 10-20 years, we, humans, are successful in creating an AGI, that is, an AI with human-level intelligence and emotions.
  
  At that point, could we destroy an AGI without its consent? Would this be considered murder?
  
  **Tags:** philosophy, agi, ethics
  
  > **Comments:**
  > **username_1**: the picture is misleading: having large amounts of compute is not equivalent to being "intelligent"  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2016-08-04. Score: 6
  
  Firstly, an AGI could conceivably exhibit all of the observable properties of intelligence without being conscious. Although that may seem counter-intuitive, at present we have no physical theory that allows us to detect consciousness (philosophically speaking, a ['Zombie'](http://plato.stanford.edu/entries/zombies/) is indistinguishable from a non-Zombie - see the writing of [Daniel Dennett](http://rads.stackoverflow.com/amzn/click/0316180661) and [David Chalmers](http://consc.net/books/tcm/) for more on this). Destroying a non-conscious entity has the same moral cost as destroying a chair.
  
  Also, note that 'destroy' doesn't necessarily mean the same for entities with *persistent substrate*, i.e. their 'brain state' can be reversibly serialized to some other storage medium and/or multiple copies of them can co-exist. So if by 'destroy' we simply mean 'switch off', then an AGI might conceivably be reassured of a subsequent re-awakening. Douglas Hofstadter gives an interesting description of such an 'episodic consciousness' in ["A Conversation with Einstein's Brain"](http://themindi.blogspot.co.uk/2007/02/chapter-26-conversation-with-einsteins.html)
  
  If by 'destroy', we mean 'irrevocably erase with no chance of re-awakening', then (unless we have a physical test which proves it is *not* conscious) destroying an entity with a seemingly human-level awareness is clearly morally tantamount to murder. To believe otherwise would be *substrate-ist* - a moral stance which may one day be seen as antiquated as racism.
  
  > **Comments:**
  > **username_4**: pzombies are kind of a debated thing though; it's rather weird to say that you can know that an AGI has no consciousness if, following Chalmers words, there is no difference in the actions taken by zombies and people; if it is unethical to kill other people, but you cannot verify if they are pzombie or person, how is it ethical to kill an AGI? It seems you come to the same conclusion in your answer though, after rereading, so I'm gonna upvote.  
  > **username_2**: @username_4 - We're actually in agreement (like Dennett, I happen to think that Zombies are logically impossible). If an AGI appears indistinguishable from a human, it would be unethical to kill it unless you can prove it has no consciousness. No-one knows what such a proof might look like, or even if one exists.  
  > **username_3**: Great answer, although I will posit that the act of destruction, even with an non-conscious entity, is not necessarily neutral from a subjective perspective.  I thinking about ideas like "intention" in a Buddhist context.  I find that, as I get older and, ostensibly, become more "evolved", I have become more and more reluctant to rack up civilian casualties in games like Grand Theft Auto. Is this is merely a form of anthropomorphism, or is there some underlying, moral imperative?  
  
  ---
  
  ### Answer by username_5 on 2016-12-02. Score: 1
  
  Even if machines with true Artificial General Intelligence were created, their apparent intelligence would still be by definition *artificial*. The word *simulation* is a [synonym](http://www.thesaurus.com/browse/artificial) and could be used to redefine AGI as Simulated General Intelligence.
  
  Keeping that in mind, **a machine that appears to be expressing emotions would only be the result of a series of complicated algorithms** allowing a computer to assess the situation and respond in an intellectually appropriate manner based on external stimulus and conditions. Every possible action this machine could possibly make would be derived from a list of possible actions the machine is capable of, no matter how large the number of possible actions grows. The machine is still a series sensors, programmed instructions, and cycles of execution.
  
  Destroying such a machine could potentially be the destruction of property if it wasn't owned by the person who destroyed it, but would it be murder? No.
  
  A broken machine can potentially be rebuilt and reactivated if it is broken. It never really died; it was destroyed. A living being that is killed is really *dead* and cannot be rebuilt and made alive once again. These key differences lead me to agree with the previous answer and conclude that **no, destroying an artificial intelligence without its consent would not be murder**.
  
  
  ---
  
  ### Answer by username_6 on 2023-04-12. Score: 0
  
  Yes we can, just like we sentence living human to death. Moral or not depends on the particular time point we are in history.
  
  
  ---
  
  ### Answer by username_7 on 2022-08-22. Score: 0
  
  There is another problem besides consciousness, which is personal identity.
  
  Consider an AI whose task is learning the discourse of a particular person during years, so as to simulate their conversation as closely as possible. After the death of the seed human, their artificial continuation can keep chatting with other humans (especially family members) using the ideas taught by the now dead person.
  
  There is a strong link between such an intelligence and a human being, so it poses a problem harder than deleting a general chatbot with no individual attachment.
  
  
  ---
  
  