Body: <p>The feedback given by humans to align artificial intelligence is limited by the reaction time and processing speed of the finite number of us, now less than <span class="math-container">$2^{33}$</span>.  As an artificial intelligence (or a growing number of them) grows in complexity, more and more possible actions need to be checked for human compatibility. But human feedback cannot grow at the pace of that expanding machine, therefore necessarily weakening the coupling between natural and artificial intelligence.</p>
  <p>Does this make alignment impossible in practice?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-08T23:12:18.637Z
Comments.Text: [A narrow model aligns the bigger model?](https://ai.stackexchange.com/q/39158/43559)
Comments.UserDisplayName: Vepir
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-09T03:15:08.297Z
Comments.Text: Then the bigger model cannot develop the ability to perform any new action whose fitness is not evaluated beforehand.
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-09T03:19:14.743Z
Comments.Text: In other words, our values exist -if at all- with respect to current options. We do not have preferences yet for future situations made possible by future systems. And the speed of appearance of those new scenarios will be faster than our speed for providing any meaningful feedback.
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-09T03:29:44.540Z
Comments.Text: In particular, are we ready for AI to beat us at the language game too, after having been surpassed at chess and go?
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-10T05:16:52.913Z
Comments.Text: Indeed it has the potential to have intelligence explosion out of control by original design, thus it needs to have some strict formal proof for the safety of self-growth or modification (like a self-replicated cellular automaton in Conway's Games of Life) limiting any bad effect, ad hoc engineering is perhaps dangerous here.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-10T05:26:57.247Z
Comments.Text: I cannot imagine but unsolvable problems. Here is one. The action of an AI system leads humans to cease direct human-human interaction. A fraction of humans is ok with that and a fraction is not. But the AI is stronger and has the effect actually implemented in reality. Even worse, that halting is an unexpected side effect. How do formal proofs deal with such questions?
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-10T23:27:30.140Z
Comments.Text: That seems a billion dollar question since the original AI needs to prove something about a future more intelligent evolution of self, like a child needs to prove the safety of adult version while at the same time no idea about all the possible versions exhaustively and even whether enumerable. This intuitively needs some perfectly reliable self-reflection. I’d suggest first try to formalize the utility of non-halting human-human interaction and prove it’s better than human-machine interaction based on some universally accepted axioms even for your two fractions of people described above.
Comments.UserDisplayName: cinch
Tags: <value-alignment>
Title: Does human attention finitude make impossible to control an expanding AI?
UserDisplayName: Jaume Oliver Lafont
ThreadText: # Does human attention finitude make impossible to control an expanding AI?
  
  (Asked by: username_0 on 2023-04-08)
  
  The feedback given by humans to align artificial intelligence is limited by the reaction time and processing speed of the finite number of us, now less than $2^{33}$. As an artificial intelligence (or a growing number of them) grows in complexity, more and more possible actions need to be checked for human compatibility. But human feedback cannot grow at the pace of that expanding machine, therefore necessarily weakening the coupling between natural and artificial intelligence.
  
  Does this make alignment impossible in practice?
  
  **Tags:** value-alignment
  
  > **Comments:**
  > **username_1**: [A narrow model aligns the bigger model?](https://ai.stackexchange.com/q/39158/43559)  
  > **username_0**: Then the bigger model cannot develop the ability to perform any new action whose fitness is not evaluated beforehand.  
  > **username_0**: In other words, our values exist -if at all- with respect to current options. We do not have preferences yet for future situations made possible by future systems. And the speed of appearance of those new scenarios will be faster than our speed for providing any meaningful feedback.  
  > **username_0**: In particular, are we ready for AI to beat us at the language game too, after having been surpassed at chess and go?  
  > **username_2**: Indeed it has the potential to have intelligence explosion out of control by original design, thus it needs to have some strict formal proof for the safety of self-growth or modification (like a self-replicated cellular automaton in Conway's Games of Life) limiting any bad effect, ad hoc engineering is perhaps dangerous here.  
  > **username_0**: I cannot imagine but unsolvable problems. Here is one. The action of an AI system leads humans to cease direct human-human interaction. A fraction of humans is ok with that and a fraction is not. But the AI is stronger and has the effect actually implemented in reality. Even worse, that halting is an unexpected side effect. How do formal proofs deal with such questions?  
  > **username_2**: That seems a billion dollar question since the original AI needs to prove something about a future more intelligent evolution of self, like a child needs to prove the safety of adult version while at the same time no idea about all the possible versions exhaustively and even whether enumerable. This intuitively needs some perfectly reliable self-reflection. I’d suggest first try to formalize the utility of non-halting human-human interaction and prove it’s better than human-machine interaction based on some universally accepted axioms even for your two fractions of people described above.  
  
  ---
  
  ## 0 answers
  
  