Answers.Body: <p>If the main goal of AI (which I assume you mean an AGI) is to protect humans and AI will be effective, then AI will always attempt to pursue its main goal (otherwise the assumption of its effectiveness does not hold), even at the expense of other less important goals that it might have. However, if the destruction of a human (or a group of humans) protected or avoided the destruction of other humans, then AI would face a dilemma. In that case, I think it is hard to predict the actions of the AI. Will it act rationally or irrationally? What would it mean for the AI to act rationally? Which parameters will it take into account? Only the number of deaths, or will take into account the future and weight the importance of the lives? How will it define the importance of a human life?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T11:58:17.960Z
Answers.Text: I think AI will go for the less damage as possible for humans because when it comes to protecting it means not to harm and if it harms humans AI will go against his will(main goal) and in that case maybe controlling man's will is the best option to not harm himself.
Answers.UserDisplayName: SgerS1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T11:59:24.367Z
Answers.Text: @giorgircheulishvili But the problem here now is how you define "damage" (as I roughly mention in my answer: which parameters will AI take into account when facing a dilemma?).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T12:09:22.827Z
Answers.Text: "
  Less damage for humans"
  as you mentioned before means AI will always attempt to pursue its main goal at the cost of other less important goals and I think less important things for AI are human living environment and his opinion about a situation.
Answers.UserDisplayName: SgerS1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T14:42:40.837Z
Answers.Text: Either the AGI is using its own intelligence to decide about goals themselves and there is no telling what it may pick. Or there are hard-coded goals and there are no dilemmas: just follow the rules.
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T15:03:35.110Z
Answers.Text: @MathieuBouville I don't think these are the only two options. There's no proof that if an AI has an hard-coded goal, then everything will be deterministic (or that there will be no dilemmas). Anyway, in my answer, I assume that an AI has the main goal of protecting humans (if this is hard-coded or not, it is not important).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T11:17:45.360Z
Answers.LastActivityDate: 2019-03-30T11:22:50.817Z
Answers.LastEditDate: 2019-03-30T11:22:50.817Z
Answers.UserDisplayName: nbro
Answers.Body: <p>If the AI is static (heuristic and fixed), it will always pursue the stated goal.  However, such a system would be "brittle", and either break or produce bad output if confronted with input not previously defined, or outside its model.</p>
  
  <p>If the AI evolves via learning, even where the goal is specific, its interpretation of that goal might change, and produce unexpected results.  (The "I, Robot" scenario.) </p>
  
  <p>If the AI is <a href="https://en.wikipedia.org/wiki/Emergence" rel="nofollow noreferrer">emergent</a>, by which I mean it evolves in way that cannot be predicted, it might evolve new goals.  </p>
  
  <p>To answer the question directly:</p>
  
  <p>Hypothetically, if there was an AGI or artificial superintelligence, or ultraintelligent machine tasked with protecting humans, and that AI perceived humans to be destroying themselves, that AI would, if able, take control of human society.  (I don't see this as contradicting its goal.)</p>
  
  <p>However, it must be stated that, in a condition of <a href="https://en.wikipedia.org/wiki/Perfect_information" rel="nofollow noreferrer">imperfect</a> &amp; <a href="https://en.wikipedia.org/wiki/Complete_information" rel="nofollow noreferrer">incomplete</a> information, where the problem is <a href="https://en.wikipedia.org/wiki/Computational_complexity_theory#Intractability" rel="nofollow noreferrer">intractable</a>, the AI is just guessing like we humans do, even if it makes better guesses, as in the case of narrowly intelligent AIs like AlphaGo.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T02:05:52.393Z
Answers.Text: About the deletion: Many thing (even for most thing that you can quote) about AI is based on assumption and it can never be verified in our life or even the lifetime of the human race, if such thing are not allowed, then the question like this should be deleted too since it can never be answered based on verified trueth
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T19:19:03.333Z
Answers.Text: @jw_ if you want to edit your answer to qualify the unproven assertions, it could be reopened.  As it was written, it causes harm.  "This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." is an unvalidated assertion. Should be presented as "consciousness **may require...**)
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T19:20:18.150Z
Answers.Text: @jw_ "The only way to realize AI and motion is use bio engineering to design new species" is incredibly problematic, because there is legitimate AI that exists, just that it's narrow.  We also have robots that produce utility, not based on bio engineering.)
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T01:40:09.087Z
Answers.Text: "This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." This is really wrong, you can check my another post after this one (hope don't delete it since its full of such staff...) - consciousness can run on any circuits in theory, but non-bio consciousness can never be economic compared with bio based. Comparing 250W 900mm2*1um AI die to 10W 1.5L human brain with self repair and reform, the former will be always nothing but a joke. And Elon musk and Hawking warn about thread of AI? Joke too.
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T01:45:01.843Z
Answers.Text: "because there is legitimate AI that exists" In my another post, it mentioned that such "AI" is just some algorithms, it should not have been called AI at all. Anyway this post is indeed not good. But the deletion policy in StackExchange is bad, down votes is enough to identify the problematic one, and consumer of this site should not be that easy affected by bad post, but deletion will erase the right information too and the OP author's work is wasted.
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-04T02:12:44.193Z
Answers.Text: BTW Celebrities worshiped by billions of people warning about the threat from will-never-mass-produced-non-bio-based-AI is the real threat and harm since it affect voters and the politicians who commonly worship them too e.g. Elon Musk is a former consultant for Trump. What about bio-based? Will ourself be called a threat since the only difference between us and the bio-based-AI is that we are designed by nature and they are designed by us? And long before that is designed successully, human race could have break down due to amatuer designed fatal microorganism.
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-07T00:52:34.210Z
Answers.Text: That's reasonable, didn't know that before.
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-23T08:04:14.747Z
Answers.Text: Now Trump compare Musk to Edison. A man in great power without basic science knowledge calls a great actor Edison, this is when the threat comes.
Answers.UserDisplayName: jw_
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-02T21:45:06.870Z
Answers.LastActivityDate: 2020-01-02T21:45:06.870Z
Answers.UserDisplayName: DukeZhou
Body: <p>If the AI goal is to serve humans and protect them (if this ever happens) and AI someday realizes that humans destroy themselves, will it try to control people for their own good, that is, will it control man's will to not destroy himself?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-30T18:20:56.097Z
Comments.Text: this is the very premise of the movie [I, Robot](https://en.wikipedia.org/wiki/I,_Robot_(film)) tangentially based on Isaac Asimov's eponymous book. In the movie it turns out that 
  >! yes, the robots take over humans on pretext of protecting humans from destroying themselves
Comments.UserDisplayName: solyarist
Tags: <philosophy><agi><ethics><ai-safety>
Title: If the AI goal is the protection of humans, will it always pursue this goal?
UserDisplayName: SgerS1
ThreadText: # If the AI goal is the protection of humans, will it always pursue this goal?
  
  (Asked by: username_0 on 2019-03-30)
  
  If the AI goal is to serve humans and protect them (if this ever happens) and AI someday realizes that humans destroy themselves, will it try to control people for their own good, that is, will it control man's will to not destroy himself?
  
  **Tags:** philosophy, agi, ethics, ai-safety
  
  > **Comments:**
  > **username_1**: this is the very premise of the movie [I, Robot](https://en.wikipedia.org/wiki/I,_Robot_(film)) tangentially based on Isaac Asimov's eponymous book. In the movie it turns out that  >! yes, the robots take over humans on pretext of protecting humans from destroying themselves  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-03-30. Score: 1
  
  If the main goal of AI (which I assume you mean an AGI) is to protect humans and AI will be effective, then AI will always attempt to pursue its main goal (otherwise the assumption of its effectiveness does not hold), even at the expense of other less important goals that it might have. However, if the destruction of a human (or a group of humans) protected or avoided the destruction of other humans, then AI would face a dilemma. In that case, I think it is hard to predict the actions of the AI. Will it act rationally or irrationally? What would it mean for the AI to act rationally? Which parameters will it take into account? Only the number of deaths, or will take into account the future and weight the importance of the lives? How will it define the importance of a human life?
  
  > **Comments:**
  > **username_0**: I think AI will go for the less damage as possible for humans because when it comes to protecting it means not to harm and if it harms humans AI will go against his will(main goal) and in that case maybe controlling man's will is the best option to not harm himself.  
  > **username_2**: @giorgircheulishvili But the problem here now is how you define "damage" (as I roughly mention in my answer: which parameters will AI take into account when facing a dilemma?).  
  > **username_0**: " Less damage for humans" as you mentioned before means AI will always attempt to pursue its main goal at the cost of other less important goals and I think less important things for AI are human living environment and his opinion about a situation.  
  > **username_3**: Either the AGI is using its own intelligence to decide about goals themselves and there is no telling what it may pick. Or there are hard-coded goals and there are no dilemmas: just follow the rules.  
  > **username_2**: @username_3 I don't think these are the only two options. There's no proof that if an AI has an hard-coded goal, then everything will be deterministic (or that there will be no dilemmas). Anyway, in my answer, I assume that an AI has the main goal of protecting humans (if this is hard-coded or not, it is not important).  
  
  ---
  
  ### Answer by username_4 on 2020-01-02. Score: 1
  
  If the AI is static (heuristic and fixed), it will always pursue the stated goal. However, such a system would be "brittle", and either break or produce bad output if confronted with input not previously defined, or outside its model.
  
  If the AI evolves via learning, even where the goal is specific, its interpretation of that goal might change, and produce unexpected results. (The "I, Robot" scenario.)
  
  If the AI is [emergent](https://en.wikipedia.org/wiki/Emergence), by which I mean it evolves in way that cannot be predicted, it might evolve new goals.
  
  To answer the question directly:
  
  Hypothetically, if there was an AGI or artificial superintelligence, or ultraintelligent machine tasked with protecting humans, and that AI perceived humans to be destroying themselves, that AI would, if able, take control of human society. (I don't see this as contradicting its goal.)
  
  However, it must be stated that, in a condition of [imperfect](https://en.wikipedia.org/wiki/Perfect_information) & [incomplete](https://en.wikipedia.org/wiki/Complete_information) information, where the problem is [intractable](https://en.wikipedia.org/wiki/Computational_complexity_theory#Intractability), the AI is just guessing like we humans do, even if it makes better guesses, as in the case of narrowly intelligent AIs like AlphaGo.
  
  > **Comments:**
  > **username_5**: About the deletion: Many thing (even for most thing that you can quote) about AI is based on assumption and it can never be verified in our life or even the lifetime of the human race, if such thing are not allowed, then the question like this should be deleted too since it can never be answered based on verified trueth  
  > **username_4**: @username_5 if you want to edit your answer to qualify the unproven assertions, it could be reopened.  As it was written, it causes harm.  "This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." is an unvalidated assertion. Should be presented as "consciousness **may require...**)  
  > **username_4**: @username_5 "The only way to realize AI and motion is use bio engineering to design new species" is incredibly problematic, because there is legitimate AI that exists, just that it's narrow.  We also have robots that produce utility, not based on bio engineering.)  
  > **username_5**: "This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." This is really wrong, you can check my another post after this one (hope don't delete it since its full of such staff...) - consciousness can run on any circuits in theory, but non-bio consciousness can never be economic compared with bio based. Comparing 250W 900mm2*1um AI die to 10W 1.5L human brain with self repair and reform, the former will be always nothing but a joke. And Elon musk and Hawking warn about thread of AI? Joke too.  
  > **username_5**: "because there is legitimate AI that exists" In my another post, it mentioned that such "AI" is just some algorithms, it should not have been called AI at all. Anyway this post is indeed not good. But the deletion policy in StackExchange is bad, down votes is enough to identify the problematic one, and consumer of this site should not be that easy affected by bad post, but deletion will erase the right information too and the OP author's work is wasted.  
  > **username_5**: BTW Celebrities worshiped by billions of people warning about the threat from will-never-mass-produced-non-bio-based-AI is the real threat and harm since it affect voters and the politicians who commonly worship them too e.g. Elon Musk is a former consultant for Trump. What about bio-based? Will ourself be called a threat since the only difference between us and the bio-based-AI is that we are designed by nature and they are designed by us? And long before that is designed successully, human race could have break down due to amatuer designed fatal microorganism.  
  > **username_5**: That's reasonable, didn't know that before.  
  > **username_5**: Now Trump compare Musk to Edison. A man in great power without basic science knowledge calls a great actor Edison, this is when the threat comes.  
  
  ---
  
  