Answers.Body: <p>Malware viruses are a very simple form of AI.  It is not difficult to conceive of a form of malware that A) can't be detected easily, B) is redundantly distributed across thousands of computers that occasionally connect to the internet, C) is capable of detecting some kinds of threats to itself and mutating to avoid the threats.</p>
  
  <p>So, simply "turning off" a malicious AI will not always be possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:27:00.447Z
Answers.LastActivityDate: 2020-01-17T21:27:00.447Z
Answers.UserDisplayName: S. McGrew
Answers.Body: <p><strong>No.</strong></p>
  
  <p>With a sufficiently advanced general AI, we can not generally assume that we can switch it off when it becomes dangerous dangerous.</p>
  
  <p>It seems that the electrical energy supply can always be switched off.  </p>
  
  <p>While that is true on the physical level, it is not guaranteed to work in practice.<br>
  The AI could cooperate with humans, which protect the AI from deactivation, or reactivate it.</p>
  
  <p>The cooperation could, for example, one of the following forms: </p>
  
  <ul>
  <li>The AI prepares a relation with humans offering a bribe for help.
  
  <ul>
  <li>An operator gets a reward for keeping the AI running, or reactivating it.</li>
  </ul></li>
  <li>The AI prepares an extortion with humans 
  
  <ul>
  <li>For example by organizing a process that needs to be actively kept running by the AI to avoid the death of a child of an operator.</li>
  </ul></li>
  <li>The AI maintains contact with a criminal organisation with strong interest to keep it running, and week ethical inhibition.
  
  <ul>
  <li>The cooperation could be with traditional organized crime organisations, providing money laundering and other vital services to a mafia organisation, which can activate it's widespread influence to maintain the AI running even when there is strong public interest in shutting it down.</li>
  </ul></li>
  </ul>
  
  <p>Note that these examples can protect against physical attacks by law enforcement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:09:28.103Z
Answers.Text: Hi and welcome to this community! I like your question and it is an important one. However, I think that your arguments in this answer are unsatisfactory. What makes you think that we wouldn't be able to stop the AI in a similar way that we can stop any other human from committing a crime?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:27:14.947Z
Answers.Text: @nbro We do not stop humans from committing crimes. We ask a human to follow laws. These laws would stop crimes if they are respected, but a human may choose to ignore them. The AI may choose to ignore it's rules. As long as an AI follows a sufficiently strict set of rules without evading them by loopholes, I do not consider it malicious.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:34:33.043Z
Answers.Text: "We do not stop humans from committing crimes.", this is, of course, a false assumption.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:39:32.040Z
Answers.Text: @nbro Yes, that's right. The case is that the AI is not cooperating, so I assume that the human also does not cooperate. We prevent some crimes by making them impossible, as when locking gemstones in a safe. Do you mean something like that?
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:47:07.963Z
Answers.Text: We can prevent certain crimes by anticipating the plan of the criminal and catch him/her. The laws exist to determine the legality of the actions.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T21:00:48.577Z
Answers.LastActivityDate: 2020-01-17T21:00:48.577Z
Answers.UserDisplayName: Volker Siegel
Body: <p>Let us assume we have a general <strong>AI that can improve itself</strong> and is at least as intelligent as humans.</p>
  
  <p>It has wide access to technical systems including the internet, and it can communicate with humans.</p>
  
  <p>The AI could become <strong>malicious</strong>. </p>
  
  <p><strong>Can we just switch off a rouge AI?</strong></p>
  
Tags: <philosophy><agi>
Title: Can we just switch off a malicious artificial intelligence?
UserDisplayName: Volker Siegel
ThreadText: # Can we just switch off a malicious artificial intelligence?
  
  (Asked by: username_0 on 2020-01-17)
  
  Let us assume we have a general **AI that can improve itself** and is at least as intelligent as humans.
  
  It has wide access to technical systems including the internet, and it can communicate with humans.
  
  The AI could become **malicious**.
  
  **Can we just switch off a rouge AI?**
  
  **Tags:** philosophy, agi
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-01-17. Score: 1
  
  Malware viruses are a very simple form of AI. It is not difficult to conceive of a form of malware that A) can't be detected easily, B) is redundantly distributed across thousands of computers that occasionally connect to the internet, C) is capable of detecting some kinds of threats to itself and mutating to avoid the threats.
  
  So, simply "turning off" a malicious AI will not always be possible.
  
  
  ---
  
  ### Answer by username_0 on 2020-01-17. Score: 0
  
  **No.**
  
  With a sufficiently advanced general AI, we can not generally assume that we can switch it off when it becomes dangerous dangerous.
  
  It seems that the electrical energy supply can always be switched off.
  
  While that is true on the physical level, it is not guaranteed to work in practice.  
  The AI could cooperate with humans, which protect the AI from deactivation, or reactivate it.
  
  The cooperation could, for example, one of the following forms:
  
  * The AI prepares a relation with humans offering a bribe for help.
    + An operator gets a reward for keeping the AI running, or reactivating it.
  * The AI prepares an extortion with humans
    + For example by organizing a process that needs to be actively kept running by the AI to avoid the death of a child of an operator.
  * The AI maintains contact with a criminal organisation with strong interest to keep it running, and week ethical inhibition.
    + The cooperation could be with traditional organized crime organisations, providing money laundering and other vital services to a mafia organisation, which can activate it's widespread influence to maintain the AI running even when there is strong public interest in shutting it down.
  
  Note that these examples can protect against physical attacks by law enforcement.
  
  > **Comments:**
  > **username_2**: Hi and welcome to this community! I like your question and it is an important one. However, I think that your arguments in this answer are unsatisfactory. What makes you think that we wouldn't be able to stop the AI in a similar way that we can stop any other human from committing a crime?  
  > **username_0**: @username_2 We do not stop humans from committing crimes. We ask a human to follow laws. These laws would stop crimes if they are respected, but a human may choose to ignore them. The AI may choose to ignore it's rules. As long as an AI follows a sufficiently strict set of rules without evading them by loopholes, I do not consider it malicious.  
  > **username_2**: "We do not stop humans from committing crimes.", this is, of course, a false assumption.  
  > **username_0**: @username_2 Yes, that's right. The case is that the AI is not cooperating, so I assume that the human also does not cooperate. We prevent some crimes by making them impossible, as when locking gemstones in a safe. Do you mean something like that?  
  > **username_2**: We can prevent certain crimes by anticipating the plan of the criminal and catch him/her. The laws exist to determine the legality of the actions.  
  
  ---
  
  