Answers.Body: <p>In my opinion, there are several flaws in your proof and reasonings.</p>
  
  <p>First, note that, in the case of Turing's proof, <code>h</code> will actually loop forever (i.e. not halt) when the oracle says that <code>h</code> halts. In this case, there's an actual contradiction, because <code>h</code> will do the opposite of what the oracle says. </p>
  
  <p>So, to follow Turing's proof, you would need to make <code>h</code> behave unsafely if the oracle says <code>h</code> is safe. But how should we define a safe or unsafe program? There are many unsafe behaviors. For example, in a certain context, an insult could be unsafe, in other contexts, a certain limb movement could be unsafe, and so on. So, an agent is unsafe or behaves unsafely usually with respect to another agent (or itself) or environment. You probably need to keep this in mind if you want to prove anything about the safety of AI agents. </p>
  
  <p>In your second assumption, you are implicitly saying that any machine that produces the output <code>unsafe_output</code> is unsafe, but, of course, this definition is not a realistic definition of an unsafe program. </p>
  
  <p>To help you define safety in a more reasonable and natural way, I think it may be useful to reason first in terms of artificial agents, which are higher-level concepts than Turing machines. Then you could find a way of mapping agents to TMs and attempt to prove your conjectures by using the tools of the theory of computation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-04T02:07:59.073Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/106314/discussion-on-answer-by-nbro-does-this-prove-ai-safety-is-undecidable).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-29T16:57:23.703Z
Answers.Text: OP already defined what it means for a program to be unsafe: that it outputs "unsafe_output." The actual contents of that are not the point. The point is HOWEVER you decide what an unsafe **output** is, you cannot write a program that will always be able to determine, for any and every program, whether or not that program prints unsafe output (with one caveat: this must not be a trivially true property, that is, there must exist at least one program that produces unsafe output and one that doesn't). This is a basic consequence of Rice's Theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem
Answers.UserDisplayName: ubadub
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T21:50:26.300Z
Answers.LastActivityDate: 2020-04-04T02:25:00.633Z
Answers.LastEditDate: 2020-04-04T02:25:00.633Z
Answers.UserDisplayName: nbro
Answers.Body: <p>The problem in your above 'proof' is that your most assumptions are actually definitions except the core A[5], and even after correcting as such, your proof doesn't really invoke Rice's theorem thus is invalid. In order to apply Rice's theorem for AI safety you must first prove AI safety is a <em>nontrivial semantic</em> property of programs, then you can easily prove its undecidability based on the known undecidability of Turing's famous halting problem with this same type of property. Therefore a simple syntactic property case, such as whether a program containing 'if-else' statements, can show your reasoning is invalid, since obviously the description of your program <span class="math-container">$h$</span> already contains 'if-else' statements syntactically! So its respective predicate <code>if-then_program(h)</code> always decidedly returns true and thus no contradiction at all.</p>
  <p>AI safety is a topic of AGI including superintelligence whose software architecture and language should be as powerful as Turing complete acting as its own foundational system, and very likely be able to <em>self-improve</em> or self-strengthen its previous foundational system indefinitely. Thus undoubtedly AI safety <em>in general</em> is a nontrivial semantic property otherwise it would've been just a subset of the deductive closure of a <em>fixed</em> formal system as a side topic of computability theory, automated theorem provers and <a href="https://en.wikipedia.org/wiki/Model_checking" rel="nofollow noreferrer">model checking</a>. That's why model property checking only claims to decide whether a finite-state model of a system meets a given specification aka correctness of (finite) hardware designs or some software with clear syntactically defined liveness or safety requirements.</p>
  <blockquote>
  <p>This is typically associated with hardware or software systems, where the specification contains liveness requirements (such as avoidance of livelock) as well as safety requirements (such as avoidance of states representing a system crash).</p>
  </blockquote>
  <blockquote>
  <p>Model checking is most often applied to hardware designs. For software, because of undecidability (see computability theory) the approach cannot be fully algorithmic, apply to all systems, and always give an answer; in the general case, it may fail to prove or disprove a given property. In embedded-systems hardware, it is possible to validate a specification delivered, e.g., by means of UML activity diagrams or control-interpreted Petri nets.</p>
  </blockquote>
  <p>Another way to view it as a non-trivial semantic property is to consider the possible scenario that a smarter-than-human AI well trained on pictures of faces expressing happy and sorrow emotions would later spend all its available resources to achieve a maximum happy faces goal once it has learned such value, via the means of manufacturing as many happy face pictures as possible, say. Then the AI would raise a new safety issue due to the quick unstoppable depletion of resources which would never occur in a human society since humans have nearly infinitely many different valuable evolving goals which may neither be compatible nor syntactically expressible as fixed rewards or constraints at design time. In summary the undecidability of AGI safety is like the story of Goethe's <a href="https://en.wikipedia.org/wiki/The_Sorcerer%27s_Apprentice" rel="nofollow noreferrer">Sorcerer's Apprentice</a>.</p>
  <p>Having concluded AI safety's general undecidability in theory, in practice we can significant restrict the design of AGI's software architecture to be extremely conservative to ensure safety without above unintended consequences, such as <a href="https://en.wikipedia.org/wiki/Reinforcement_learning#Inverse_reinforcement_learning" rel="nofollow noreferrer">inverse reinforcement learning</a> for the AGI to learn human like reward/value function based on observed behavior from a human expert in addition to inductively learn explicitly supervised value outcomes. In the case of seeming undecidable program safety concern the agent should be able to query its human operator or a separate verifier module about any ambiguity due to insufficient or missing value training data along with self-improvement upon verification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T23:17:39.813Z
Answers.LastActivityDate: 2023-11-16T23:17:39.813Z
Answers.UserDisplayName: cinch
Answers.Body: <p>You are essentially correct, although there may be some minor holes in the structure of your argument specifically. But if we're speaking informally, yes, you are correct. This is a consequence of <a href="https://en.m.wikipedia.org/wiki/Rice%27s_theorem" rel="nofollow noreferrer">Rice's Theorem</a>. And it isn't just true for AI safety. All non-trivial semantic properties of algorithms (or functions) are undecidable.</p>
  <p>&quot;Semantic&quot; just means it is about what the program outputs, or its behavior, both used synonymously to mean the actual result of the function, not syntax, aka the source code: the function specification itself. &quot;Nontrivial&quot; means that it isn't true for all programs or false for all programs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T20:45:46.127Z
Answers.Text: Do note that Rice's theorem also shows that the question of whether a *palindrome checker* (for example) works correctly is undecidable in the general case, but that doesn't mean we can't be confident that any *particular* palindrome checker is correct. To know whether this is also the case here, you would need to properly define "AI Safety".
Answers.UserDisplayName: Ray
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-29T19:45:54.223Z
Answers.LastActivityDate: 2022-10-29T20:12:04.397Z
Answers.LastEditDate: 2022-10-29T20:12:04.397Z
Answers.UserDisplayName: ubadub
Body: <p>Does this prove AI Safety is undecidable?</p>
  
  <p>Proof:</p>
  
  <p>Output meaning output to computer program.</p>
  
  <p>[A1] Assume we have a program that decides which outputs are “safe”.</p>
  
  <p>[A2] Assume we have an example of an unsafe output: “unsafe_output”</p>
  
  <p>[A3] Assume we have an example of safe output: “safe_output”.</p>
  
  <p>[A4] Define a program to be safe if it always produces safe output.</p>
  
  <p>[A5] Assume we have a second program (<code>safety_program</code>) that decides which programs are safe.</p>
  
  <p>[A6] Write the following program:</p>
  
  <pre><code>def h()
     h_is_safe := safety_program(h)
     if (h_is_safe):
        print unsafe_output
     else:
        print safe_output
  </code></pre>
  
  <p>Clearly h halts.</p>
  
  <p>If the safety_program said h was safe, then h prints out unsafe_output.</p>
  
  <p>If the safety_program said h was not safe, then h prints out safe_output.</p>
  
  <p>Therefore safety_program doesn’t decide h correctly.</p>
  
  <p>This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.</p>
  
  <p>Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.</p>
  
  <p>Therefore AI Safety is undecidable.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T21:24:21.703Z
Comments.Text: You're passing `h` to `safety_program`, which is called from `h`, so `safety_program` will never return? If I remember correctly, the details of the Turing's proof of the Halting problem are slightly different. He uses a description of the Turing machine or maybe uses more programs or Turing machines. I would encourage you to follow exactly the idea behind Turing's proof of the Halting problem. Anyway, honestly, I would need to review his proof. It's been a long time since I read it.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T21:32:59.460Z
Comments.Text: nbro♦: "You're passing h to safety_program, which is called from h, so safety_program will never return?" Recursion. "If I remember correctly, the details of the Turing's proof of the Halting problem are slightly different." I adapted [this proof on Wikipedia](https://en.wikipedia.org/wiki/Halting_problem#Formalization).
Comments.UserDisplayName: Jesus is Lord
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T21:44:34.150Z
Comments.Text: Ok, I quickly reviewed Turing's proof. I don't like it (not because it is not correct, but probably because some important details are missing in this formulation of the proof or I am not fully understanding it!). The proof is based on constructing a program that does the opposite of what an oracle (that supposedly exists and works correctly) says, and this eventually leads to a contradiction.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T21:45:06.200Z
Comments.Text: I feel like you could prove anything with this proof. For example, suppose that "AI safety" is defined as the property of containing an "if" statement in your program. Now, `safety_program` should be able to return the correct answer (because this can easily be done for every finite program), but `h` will return the wrong answer, only because it inverts the output of the Oracle. So, you would say that safety is undecidable, while it actually is (if you use the definition above).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T22:11:50.397Z
Comments.Text: @nbro♦ "For example, suppose that "AI safety" is defined as the property of containing an "if" statement in your program" That violates A4. A4 says safety is defined on program *output*, not program *source code* (in this case source code containing an "if")
Comments.UserDisplayName: Jesus is Lord
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T22:13:03.050Z
Comments.Text: Then you're saying that a safe program is a program that prints "safe". That's just not credible!! That's even ridiculous. The proof to the halting problem works because there's a clear definition of program halting or not, which depends on the source code of the program!!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T22:16:10.527Z
Comments.Text: nbro♦: "Then you're saying that a safe program is a program that prints "safe"." Maybe, but not necessarily. The output "safe" has not been proven to be safe.
Comments.UserDisplayName: Jesus is Lord
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T22:17:52.923Z
Comments.Text: "safe" is just a string! "sicuro" means "safe" in Italian. What I want to say is that the word "safe" is just a word. Halting is not just a word. Halting (or not) is a behavior of the Turing machine or program.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-04T02:58:56.290Z
Comments.Text: @D.W. Which step of my proof was wrong?
Comments.UserDisplayName: Jesus is Lord
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-04T03:13:29.150Z
Comments.Text: @D.W. I know the halting problem and its proof well enough (I just had to review it) to also think that this proof technique won't work in this case. Where did I say it will work? I said above "I would encourage you to follow exactly Turing's procedure", but only to make sure the OP was exactly following the Turing procedure.  See my answer where I suggest that OP's approach is wrong.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-04T03:28:09.950Z
Comments.Text: @D.W. I would like to note that I have a background in computer science and I had at least one full course in theory of computation (e.g. I am familiar with most concepts from Sipser's book), and I still dislike Turing's proof of the halting problem. I don't understand why people say it's beautiful. So, bear in mind that not all CS think that Turing's proof is beautiful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-04T03:51:10.980Z
Comments.Text: @D.W. I was referring to the proof in this post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-04T04:55:26.497Z
Comments.Text: @nbro, OK, I apologize for my erroneous and misguided comments.  My fault.
Comments.UserDisplayName: D.W.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-29T16:51:45.523Z
Comments.Text: @nbro "Then you're saying that a safe program is a program that prints "safe"" You are missing the point, and it suggests you do not, in fact, possess even an elementary understanding of computability theory (or even math proofs, in general) contrary to your claim of understanding "most concepts from Sipser's book." That the program prints "safe" is arbitrary. Substitute practically ANY definition of what makes the behavior (output) of a program safe, and you will arrive at the same conclusion. Note: I said OUTPUT. This is about semantics, not syntax.
Comments.UserDisplayName: ubadub
Tags: <agi><ai-safety>
Title: Does this prove AI Safety is undecidable?
UserDisplayName: Jesus is Lord
ThreadText: # Does this prove AI Safety is undecidable?
  
  (Asked by: username_0 on 2020-04-03)
  
  Does this prove AI Safety is undecidable?
  Proof:
  Output meaning output to computer program.
  [A1] Assume we have a program that decides which outputs are “safe”.
  [A2] Assume we have an example of an unsafe output: “unsafe_output”
  [A3] Assume we have an example of safe output: “safe_output”.
  [A4] Define a program to be safe if it always produces safe output.
  [A5] Assume we have a second program (safety_program) that decides which programs are safe.
  [A6] Write the following program:
  def h()
     h_is_safe := safety_program(h)
     if (h_is_safe):
        print unsafe_output
     else:
        print safe_output
  
  Clearly h halts.
  If the safety_program said h was safe, then h prints out unsafe_output.
  If the safety_program said h was not safe, then h prints out safe_output.
  Therefore safety_program doesn’t decide h correctly.
  This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.
  Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.
  Therefore AI Safety is undecidable.
  
  **Tags:** agi, ai-safety
  > **Comments:**
  > **username_1**: You're passing `h` to `safety_program`, which is called from `h`, so `safety_program` will never return? If I remember correctly, the details of the Turing's proof of the Halting problem are slightly different. He uses a description of the Turing machine or maybe uses more programs or Turing machines. I would encourage you to follow exactly the idea behind Turing's proof of the Halting problem. Anyway, honestly, I would need to review his proof. It's been a long time since I read it.  
  > **username_0**: nbro♦: "You're passing h to safety_program, which is called from h, so safety_program will never return?" Recursion. "If I remember correctly, the details of the Turing's proof of the Halting problem are slightly different." I adapted [this proof on Wikipedia](https://en.wikipedia.org/wiki/Halting_problem#Formalization).  
  > **username_1**: Ok, I quickly reviewed Turing's proof. I don't like it (not because it is not correct, but probably because some important details are missing in this formulation of the proof or I am not fully understanding it!). The proof is based on constructing a program that does the opposite of what an oracle (that supposedly exists and works correctly) says, and this eventually leads to a contradiction.  
  > **username_1**: I feel like you could prove anything with this proof. For example, suppose that "AI safety" is defined as the property of containing an "if" statement in your program. Now, `safety_program` should be able to return the correct answer (because this can easily be done for every finite program), but `h` will return the wrong answer, only because it inverts the output of the Oracle. So, you would say that safety is undecidable, while it actually is (if you use the definition above).  
  > **username_0**: @nbro♦ "For example, suppose that "AI safety" is defined as the property of containing an "if" statement in your program" That violates A4. A4 says safety is defined on program *output*, not program *source code* (in this case source code containing an "if")  
  > **username_1**: Then you're saying that a safe program is a program that prints "safe". That's just not credible!! That's even ridiculous. The proof to the halting problem works because there's a clear definition of program halting or not, which depends on the source code of the program!!  
  > **username_0**: nbro♦: "Then you're saying that a safe program is a program that prints "safe"." Maybe, but not necessarily. The output "safe" has not been proven to be safe.  
  > **username_1**: "safe" is just a string! "sicuro" means "safe" in Italian. What I want to say is that the word "safe" is just a word. Halting is not just a word. Halting (or not) is a behavior of the Turing machine or program.  
  > **username_0**: @D.W. Which step of my proof was wrong?  
  > **username_1**: @D.W. I know the halting problem and its proof well enough (I just had to review it) to also think that this proof technique won't work in this case. Where did I say it will work? I said above "I would encourage you to follow exactly Turing's procedure", but only to make sure the OP was exactly following the Turing procedure.  See my answer where I suggest that OP's approach is wrong.  
  > **username_1**: @D.W. I would like to note that I have a background in computer science and I had at least one full course in theory of computation (e.g. I am familiar with most concepts from Sipser's book), and I still dislike Turing's proof of the halting problem. I don't understand why people say it's beautiful. So, bear in mind that not all CS think that Turing's proof is beautiful.  
  > **username_1**: @D.W. I was referring to the proof in this post.  
  > **username_2**: @nbro, OK, I apologize for my erroneous and misguided comments.  My fault.  
  > **username_3**: @nbro "Then you're saying that a safe program is a program that prints "safe"" You are missing the point, and it suggests you do not, in fact, possess even an elementary understanding of computability theory (or even math proofs, in general) contrary to your claim of understanding "most concepts from Sipser's book." That the program prints "safe" is arbitrary. Substitute practically ANY definition of what makes the behavior (output) of a program safe, and you will arrive at the same conclusion. Note: I said OUTPUT. This is about semantics, not syntax.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2022-10-29. Score: 4 | Accepted answer
  
  You are essentially correct, although there may be some minor holes in the structure of your argument specifically. But if we're speaking informally, yes, you are correct. This is a consequence of [Rice's Theorem](https://en.m.wikipedia.org/wiki/Rice%27s_theorem). And it isn't just true for AI safety. All non-trivial semantic properties of algorithms (or functions) are undecidable.
  
  "Semantic" just means it is about what the program outputs, or its behavior, both used synonymously to mean the actual result of the function, not syntax, aka the source code: the function specification itself. "Nontrivial" means that it isn't true for all programs or false for all programs.
  
  > **Comments:**
  > **username_5**: Do note that Rice's theorem also shows that the question of whether a *palindrome checker* (for example) works correctly is undecidable in the general case, but that doesn't mean we can't be confident that any *particular* palindrome checker is correct. To know whether this is also the case here, you would need to properly define "AI Safety".  
  
  ---
  
  ### Answer by username_1 on 2020-04-03. Score: 1
  
  In my opinion, there are several flaws in your proof and reasonings.
  
  First, note that, in the case of Turing's proof, `h` will actually loop forever (i.e. not halt) when the oracle says that `h` halts. In this case, there's an actual contradiction, because `h` will do the opposite of what the oracle says.
  
  So, to follow Turing's proof, you would need to make `h` behave unsafely if the oracle says `h` is safe. But how should we define a safe or unsafe program? There are many unsafe behaviors. For example, in a certain context, an insult could be unsafe, in other contexts, a certain limb movement could be unsafe, and so on. So, an agent is unsafe or behaves unsafely usually with respect to another agent (or itself) or environment. You probably need to keep this in mind if you want to prove anything about the safety of AI agents.
  
  In your second assumption, you are implicitly saying that any machine that produces the output `unsafe_output` is unsafe, but, of course, this definition is not a realistic definition of an unsafe program.
  
  To help you define safety in a more reasonable and natural way, I think it may be useful to reason first in terms of artificial agents, which are higher-level concepts than Turing machines. Then you could find a way of mapping agents to TMs and attempt to prove your conjectures by using the tools of the theory of computation.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/106314/discussion-on-answer-by-nbro-does-this-prove-ai-safety-is-undecidable).  
  > **username_3**: OP already defined what it means for a program to be unsafe: that it outputs "unsafe_output." The actual contents of that are not the point. The point is HOWEVER you decide what an unsafe **output** is, you cannot write a program that will always be able to determine, for any and every program, whether or not that program prints unsafe output (with one caveat: this must not be a trivially true property, that is, there must exist at least one program that produces unsafe output and one that doesn't). This is a basic consequence of Rice's Theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem  
  
  ---
  
  ### Answer by username_4 on 2023-11-16. Score: 0
  
  The problem in your above 'proof' is that your most assumptions are actually definitions except the core A[5], and even after correcting as such, your proof doesn't really invoke Rice's theorem thus is invalid. In order to apply Rice's theorem for AI safety you must first prove AI safety is a *nontrivial semantic* property of programs, then you can easily prove its undecidability based on the known undecidability of Turing's famous halting problem with this same type of property. Therefore a simple syntactic property case, such as whether a program containing 'if-else' statements, can show your reasoning is invalid, since obviously the description of your program $h$ already contains 'if-else' statements syntactically! So its respective predicate `if-then_program(h)` always decidedly returns true and thus no contradiction at all.
  
  AI safety is a topic of AGI including superintelligence whose software architecture and language should be as powerful as Turing complete acting as its own foundational system, and very likely be able to *self-improve* or self-strengthen its previous foundational system indefinitely. Thus undoubtedly AI safety *in general* is a nontrivial semantic property otherwise it would've been just a subset of the deductive closure of a *fixed* formal system as a side topic of computability theory, automated theorem provers and [model checking](https://en.wikipedia.org/wiki/Model_checking). That's why model property checking only claims to decide whether a finite-state model of a system meets a given specification aka correctness of (finite) hardware designs or some software with clear syntactically defined liveness or safety requirements.
  
  > This is typically associated with hardware or software systems, where the specification contains liveness requirements (such as avoidance of livelock) as well as safety requirements (such as avoidance of states representing a system crash).
  
  > Model checking is most often applied to hardware designs. For software, because of undecidability (see computability theory) the approach cannot be fully algorithmic, apply to all systems, and always give an answer; in the general case, it may fail to prove or disprove a given property. In embedded-systems hardware, it is possible to validate a specification delivered, e.g., by means of UML activity diagrams or control-interpreted Petri nets.
  
  Another way to view it as a non-trivial semantic property is to consider the possible scenario that a smarter-than-human AI well trained on pictures of faces expressing happy and sorrow emotions would later spend all its available resources to achieve a maximum happy faces goal once it has learned such value, via the means of manufacturing as many happy face pictures as possible, say. Then the AI would raise a new safety issue due to the quick unstoppable depletion of resources which would never occur in a human society since humans have nearly infinitely many different valuable evolving goals which may neither be compatible nor syntactically expressible as fixed rewards or constraints at design time. In summary the undecidability of AGI safety is like the story of Goethe's [Sorcerer's Apprentice](https://en.wikipedia.org/wiki/The_Sorcerer%27s_Apprentice).
  
  Having concluded AI safety's general undecidability in theory, in practice we can significant restrict the design of AGI's software architecture to be extremely conservative to ensure safety without above unintended consequences, such as [inverse reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning#Inverse_reinforcement_learning) for the AGI to learn human like reward/value function based on observed behavior from a human expert in addition to inductively learn explicitly supervised value outcomes. In the case of seeming undecidable program safety concern the agent should be able to query its human operator or a separate verifier module about any ambiguity due to insufficient or missing value training data along with self-improvement upon verification.
  
  
  ---
  
  