Answers.Body: <p>I like mindrime's answer as it identifies fundamental, applicable notions.  I'll attempt another short answer from a Game Theory perspective.</p>
  
  <p>Game Theory was founded in the <a href="https://en.wikipedia.org/wiki/Minimax" rel="nofollow noreferrer">minimax</a> principle.  Specifically, maximizing benefit while minimizing potential downside in a condition of uncertainty.  </p>
  
  <p>Minimax works very well in contexts with easily definable parameters, such as combinatorial games and procedural optimization, but it gets more tricky in real-world scenarios where too many parameters may be present, resulting in a <a href="https://en.wikipedia.org/wiki/Combinatorial_explosion" rel="nofollow noreferrer">combinatorial explosion</a>. </p>
  
  <p>Another issue regarding real-world applications arises from the <a href="https://en.wikipedia.org/wiki/Symbol_grounding_problem" rel="nofollow noreferrer">symbol grounding problem</a>.  In the paper clip scenario, the benefit sought (the goal) is clearly definable, and can be expressed mathematically.  By contrast, the downsides to be avoided are more difficult to define: "Don't destroy the world", "Don't use up resources to a degree that humans suffer", "Don't harm the environment" all rely on language.  This is to say they rely on terms that, at present, constitute symbols that cannot be grounded.  Thus there is room for misinterpretation that may yield unintended consequences.</p>
  
  <p>"Minimizing impact on the system" (minimizing influence in the world) would be the goal, but how do you guarantee an automata has a clear understanding of what this entails in every possible scenario?</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T21:08:34.527Z
Answers.LastActivityDate: 2017-08-23T21:13:44.180Z
Answers.LastEditDate: 2017-08-23T21:13:44.180Z
Answers.UserDisplayName: DukeZhou
Answers.Body: <p>Telling the system "minimize your interference with the world" while also telling it to "maximize paperclip production" or whatever is interesting on at least one level, and that is this: how exactly does the system quantify "interference in the world"?  That seems like an ill-defined notion offhand.  But if you could quantify it, then it just becomes one more variable in an optimization problem which is a straightforward notion.</p>
  
  <p>Detecting a runaway process in general is an interesting concept.  I am not an expert, but I am betting there is some material in the <a href="https://en.wikipedia.org/wiki/Cybernetics" rel="nofollow noreferrer">cybernetics</a> / <a href="https://en.wikipedia.org/wiki/Control_theory" rel="nofollow noreferrer">control theory</a> literature on this topic.  It might be as simple as watching some rate-of-change (paper-clips produced per day?) and take the first and second derivatives and look for sharp changes in <a href="https://en.wikipedia.org/wiki/Acceleration" rel="nofollow noreferrer">acceleration</a> or <a href="https://en.wikipedia.org/wiki/Jerk_(physics)" rel="nofollow noreferrer">jerk</a>.  Other algorithms from the world of <a href="https://en.wikipedia.org/wiki/Anomaly_detection" rel="nofollow noreferrer">anomaly detection</a> might also be applicable.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T18:47:51.900Z
Answers.LastActivityDate: 2017-08-23T18:47:51.900Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>From the <a href="https://rationalwiki.org/wiki/AI-box_experiment" rel="nofollow noreferrer">AI in a box</a> literature, it is argued that even just a text interface with the rest of the world is sufficient for an AI to gain total control. </p>
  
  <p>Or, consider the literature regarding phase state changes / dynamical systems / control theory. I don't know if there is a source that directly argues for this, but its imaginable that since societal systems are so interconnected, a few controllable free parameters of a system might be sufficient to strongly influence the system as a whole.</p>
  
  <p>So no, restricting influence is not a sufficient guarantee of reducing AI risk. A common saying is that if we know the goal of some AI, we can't predict /how/ the AI would achieve some goal since we aren't smart enough, but we can predict the eventual outcome (its success).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-28T00:27:45.610Z
Answers.LastActivityDate: 2018-01-28T00:27:45.610Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>Here you make the wrong assumption that AI will have only one goal at a time. But the same as humans it will have to have in mind many goals all the time it should follow and watch out that newly assigned goal dont conflict with its existing goals.</p>
  
  <p>Your proposition to give ai the goal "minimize impact on world" is simplistic as it would be harmful in some situations.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-28T09:49:47.373Z
Answers.LastActivityDate: 2018-01-28T09:49:47.373Z
Answers.UserDisplayName: Tone Škoda
Body: <p>Let's take our standard <a href="https://wiki.lesswrong.com/wiki/Paperclip_maximizer" rel="nofollow noreferrer">paperclip maximizer General AI</a> and attempt to obtain precisely one million paper clips, over course of a year, without destroying the universe in the process.</p>
  
  <p>Most maximization directives make the process run-away. As cheaply as possible will crash world economy. As good clips as possible will turn the universe into super-synthesizer that assembles atom-perfect paperclips. Adding a deadline on these maximization processes will probably result in terrorizing the staff into readjusting the deadline, or invention of time travel (after consuming the solar system to invent it.) Minimizing resource usage would likely result in closure of all industry world-wide. You know, the standard horror scenarios.</p>
  
  <p>What about the directive of minimizing AI's influence on the world while completing the task? Would it be safe, or can you spot a runaway scenario where it could result in dire effects?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-27T20:00:38.270Z
Comments.Text: it would also be quite diminishing the potential power of AI
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-27T21:27:18.530Z
Comments.Text: @k.c.sayz'k.csayz': The problem is the consequences for it going runaway are way worse than any loss of performance.
Comments.UserDisplayName: SF.
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-28T00:42:29.617Z
Comments.Text: i added an answer that is a response to the above
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-28T22:22:31.270Z
Comments.Text: Hannu Rajaniemi wrote an interesting story on this subject, called "[Deus Ex Homine](https://www.e-reading.club/chapter.php/71247/44/Dozois_-_The_Years_Best_Science_Fiction_23rd_Annual_Collection_%282006%29.html)"  re: benevolent superintelligences, described as recursively self-optimizing algorithms.  In this story, the superintelligences continually improve and fix things, but are ungovernable and run wild.  It's an interesting take because it looks at the scenario from the opposite angle (i.e. not superintelligence that want to destroy humanity, but improve it. :)
Comments.UserDisplayName: DukeZhou
Tags: <agi>
Title: Would minimizing influence into the world be a safe directive to a general AI?
UserDisplayName: SF.
ThreadText: # Would minimizing influence into the world be a safe directive to a general AI?
  
  (Asked by: username_0 on 2017-08-23)
  
  Let's take our standard [paperclip maximizer General AI](https://wiki.lesswrong.com/wiki/Paperclip_maximizer) and attempt to obtain precisely one million paper clips, over course of a year, without destroying the universe in the process.
  
  Most maximization directives make the process run-away. As cheaply as possible will crash world economy. As good clips as possible will turn the universe into super-synthesizer that assembles atom-perfect paperclips. Adding a deadline on these maximization processes will probably result in terrorizing the staff into readjusting the deadline, or invention of time travel (after consuming the solar system to invent it.) Minimizing resource usage would likely result in closure of all industry world-wide. You know, the standard horror scenarios.
  
  What about the directive of minimizing AI's influence on the world while completing the task? Would it be safe, or can you spot a runaway scenario where it could result in dire effects?
  
  **Tags:** agi
  > **Comments:**
  > **username_1**: it would also be quite diminishing the potential power of AI  
  > **username_0**: @k.c.sayz'k.csayz': The problem is the consequences for it going runaway are way worse than any loss of performance.  
  > **username_1**: i added an answer that is a response to the above  
  > **username_2**: Hannu Rajaniemi wrote an interesting story on this subject, called "[Deus Ex Homine](https://www.e-reading.club/chapter.php/71247/44/Dozois_-_The_Years_Best_Science_Fiction_23rd_Annual_Collection_%282006%29.html)"  re: benevolent superintelligences, described as recursively self-optimizing algorithms.  In this story, the superintelligences continually improve and fix things, but are ungovernable and run wild.  It's an interesting take because it looks at the scenario from the opposite angle (i.e. not superintelligence that want to destroy humanity, but improve it. :)  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2017-08-23. Score: 3
  
  I like mindrime's answer as it identifies fundamental, applicable notions. I'll attempt another short answer from a Game Theory perspective.
  
  Game Theory was founded in the [minimax](https://en.wikipedia.org/wiki/Minimax) principle. Specifically, maximizing benefit while minimizing potential downside in a condition of uncertainty.
  
  Minimax works very well in contexts with easily definable parameters, such as combinatorial games and procedural optimization, but it gets more tricky in real-world scenarios where too many parameters may be present, resulting in a [combinatorial explosion](https://en.wikipedia.org/wiki/Combinatorial_explosion).
  
  Another issue regarding real-world applications arises from the [symbol grounding problem](https://en.wikipedia.org/wiki/Symbol_grounding_problem). In the paper clip scenario, the benefit sought (the goal) is clearly definable, and can be expressed mathematically. By contrast, the downsides to be avoided are more difficult to define: "Don't destroy the world", "Don't use up resources to a degree that humans suffer", "Don't harm the environment" all rely on language. This is to say they rely on terms that, at present, constitute symbols that cannot be grounded. Thus there is room for misinterpretation that may yield unintended consequences.
  
  "Minimizing impact on the system" (minimizing influence in the world) would be the goal, but how do you guarantee an automata has a clear understanding of what this entails in every possible scenario?
  
  
  ---
  
  ### Answer by username_3 on 2017-08-23. Score: 2
  
  Telling the system "minimize your interference with the world" while also telling it to "maximize paperclip production" or whatever is interesting on at least one level, and that is this: how exactly does the system quantify "interference in the world"? That seems like an ill-defined notion offhand. But if you could quantify it, then it just becomes one more variable in an optimization problem which is a straightforward notion.
  
  Detecting a runaway process in general is an interesting concept. I am not an expert, but I am betting there is some material in the [cybernetics](https://en.wikipedia.org/wiki/Cybernetics) / [control theory](https://en.wikipedia.org/wiki/Control_theory) literature on this topic. It might be as simple as watching some rate-of-change (paper-clips produced per day?) and take the first and second derivatives and look for sharp changes in [acceleration](https://en.wikipedia.org/wiki/Acceleration) or [jerk](https://en.wikipedia.org/wiki/Jerk_(physics)). Other algorithms from the world of [anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) might also be applicable.
  
  
  ---
  
  ### Answer by username_1 on 2018-01-28. Score: 1
  
  From the [AI in a box](https://rationalwiki.org/wiki/AI-box_experiment) literature, it is argued that even just a text interface with the rest of the world is sufficient for an AI to gain total control.
  
  Or, consider the literature regarding phase state changes / dynamical systems / control theory. I don't know if there is a source that directly argues for this, but its imaginable that since societal systems are so interconnected, a few controllable free parameters of a system might be sufficient to strongly influence the system as a whole.
  
  So no, restricting influence is not a sufficient guarantee of reducing AI risk. A common saying is that if we know the goal of some AI, we can't predict /how/ the AI would achieve some goal since we aren't smart enough, but we can predict the eventual outcome (its success).
  
  
  ---
  
  ### Answer by username_4 on 2018-01-28. Score: 0
  
  Here you make the wrong assumption that AI will have only one goal at a time. But the same as humans it will have to have in mind many goals all the time it should follow and watch out that newly assigned goal dont conflict with its existing goals.
  
  Your proposition to give ai the goal "minimize impact on world" is simplistic as it would be harmful in some situations.
  
  
  ---
  
  