Body: <p>I was thinking about the risks of <em>Oracle AI</em> and it doesn't seem as safe to me as Bostrom et al. <a href="http://www.aleph.se/papers/oracleAI.pdf" rel="nofollow noreferrer">suggest</a>. From my point of view, even an AGI that only answers questions could have a catastrophic impact. Thinking about it a little bit, I came up with this <em>Proof</em>:</p>
  
  <h3>Lemma</h3>
  
  <blockquote>
    <p>We are not safe even by giving the oracle the ability to only answer yes or no.</p>
  </blockquote>
  
  <h3>Proof</h3>
  
  <blockquote>
    <p>Let's say that our oracle must maximize an utility function <span class="math-container">$\phi$</span>, there is a procedure that encodes the optimality of <span class="math-container">$\phi$</span>. Since a procedure is, in fact, a set of instructions (an algorithm), each procedure can be encoded as a binary string, composed solely of 0 and 1, Therefore we will have <span class="math-container">$\phi \in {\{0,1\}^n}$</span>,  assuming that the optimal procedure has finite cardinality.  Shannon's entropy tells us that every binary string can be guessed by answering only yes/no to questions like: is the first bit 0? and so on, therefore we can reconstruct any algorithm via binary answers (yes / no).</p>
  </blockquote>
  
  <p>Is this reasoning correct and applicable to this type of AI?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T19:24:44.553Z
Comments.Text: Can you clarify the relationship between your proof that uses Shannon's entropy and the fact that the oracle could be dangerous? How would the ability of guessing make an oracle dangerous?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T20:22:52.537Z
Comments.Text: My proof is simple: you can implement every procedure just by answering yes/no in the same way that you can compose every string just by answering yes/no.
Comments.UserDisplayName: Yamar69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T20:25:44.960Z
Comments.Text: How can an oracle that just guesses and cannot do anything else (e.g. move something) be dangerous? She may be used to retrieve information, etc., but, if she cannot act, apart from guessing questions, how can it be dangerous? Well, if someone uses the oracle to do something, that could be dangerous, but the oracle alone will not. What do you mean by "you can implement every procedure just by answering yes/no..." How can you compose a string by answering yes or no? Someone needs to ask the question anyway. Are you talking about any theory of computation concept (that I don't recall now)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T20:28:56.863Z
Comments.Text: It can by fooling the humans that make the questions. Of course the procedure will be exponentially longer in respect to a "free and cross domain ai" but it can nonetheles reach its final goal.
Comments.UserDisplayName: Yamar69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T20:30:52.537Z
Comments.Text: Your argument is interesting. However, I think that it misses some details (and therefore could be flawed). For example, what is your definition of an oracle? When proving something, you precisely define your terms, so that everyone knows what you're talking about. I encourage you to define the oracle.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T05:11:25.533Z
Comments.Text: @nbro How *reconstruction of an algorithm* is related to *safety*?
Comments.UserDisplayName: Abhas Kumar Sinha
Tags: <philosophy><agi><proofs><risk-management>
Title: Is an oracle that answers only with a "yes" or "no" dangerous?
UserDisplayName: Yamar69
ThreadText: # Is an oracle that answers only with a "yes" or "no" dangerous?
  
  (Asked by: username_0 on 2020-03-06)
  
  I was thinking about the risks of *Oracle AI* and it doesn't seem as safe to me as Bostrom et al. [suggest](http://www.aleph.se/papers/oracleAI.pdf). From my point of view, even an AGI that only answers questions could have a catastrophic impact. Thinking about it a little bit, I came up with this *Proof*:
  
  ### Lemma
  
  > We are not safe even by giving the oracle the ability to only answer yes or no.
  
  ### Proof
  
  > Let's say that our oracle must maximize an utility function $\phi$, there is a procedure that encodes the optimality of $\phi$. Since a procedure is, in fact, a set of instructions (an algorithm), each procedure can be encoded as a binary string, composed solely of 0 and 1, Therefore we will have $\phi \in {\{0,1\}^n}$, assuming that the optimal procedure has finite cardinality. Shannon's entropy tells us that every binary string can be guessed by answering only yes/no to questions like: is the first bit 0? and so on, therefore we can reconstruct any algorithm via binary answers (yes / no).
  
  Is this reasoning correct and applicable to this type of AI?
  
  **Tags:** philosophy, agi, proofs, risk-management
  
  > **Comments:**
  > **username_1**: Can you clarify the relationship between your proof that uses Shannon's entropy and the fact that the oracle could be dangerous? How would the ability of guessing make an oracle dangerous?  
  > **username_0**: My proof is simple: you can implement every procedure just by answering yes/no in the same way that you can compose every string just by answering yes/no.  
  > **username_1**: How can an oracle that just guesses and cannot do anything else (e.g. move something) be dangerous? She may be used to retrieve information, etc., but, if she cannot act, apart from guessing questions, how can it be dangerous? Well, if someone uses the oracle to do something, that could be dangerous, but the oracle alone will not. What do you mean by "you can implement every procedure just by answering yes/no..." How can you compose a string by answering yes or no? Someone needs to ask the question anyway. Are you talking about any theory of computation concept (that I don't recall now)?  
  > **username_0**: It can by fooling the humans that make the questions. Of course the procedure will be exponentially longer in respect to a "free and cross domain ai" but it can nonetheles reach its final goal.  
  > **username_1**: Your argument is interesting. However, I think that it misses some details (and therefore could be flawed). For example, what is your definition of an oracle? When proving something, you precisely define your terms, so that everyone knows what you're talking about. I encourage you to define the oracle.  
  > **username_2**: @username_1 How *reconstruction of an algorithm* is related to *safety*?  
  
  ---
  
  ## 0 answers
  
  