Answers.Body: <blockquote>
  <p>What if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?</p>
  </blockquote>
  <p>If this turns out to be the case, then the complexity would indeed be the bottleneck.</p>
  <p>There's a formal term for this: <a href="https://en.wikipedia.org/wiki/Kolmogorov_complexity" rel="nofollow noreferrer">kolmogrov complexity or &quot;k-complexity&quot;</a>.  (The best formal definition of intelligence I've come across is Hutter &amp; Legg's, which is broken down in the article &quot;<a href="https://nbro.github.io/blogging/2020/05/20/on-the-definition-of-intelligence/" rel="nofollow noreferrer">On the definition of intelligence</a>&quot; by our own nbro.  It will shed shed some light on the formal structure of the question at hand.</p>
  <ul>
  <li>Any attempt to provide values to your d &amp; l would speculative, since we're still so far from AGI. The question would be what is the effect of any given (d,l)</li>
  </ul>
  <p>The subject can definitely be treated formally in a mathematical sense the and values analyzed.  (There may even been some formulae out there, although, they are likely to be more complex, per the Hutter&amp;Legg.)</p>
  <p>Sorry I can't give you a more formal answer--it's a good question!</p>
  <br>
  <p>Notes:</p>
  <p>Another bottleneck would be the speed of light, essentially, &quot;bits and bytes across a pipeline&quot;.  This would be a factor in both self-replication and networked processing and memory.  You can expand bandwidth/throughput, but not infinitely.</p>
  <p>This relates to space complexity (how much volume/memory does the algorithm require?) in addition to the k-complexity, which itself is likely to be non-trivial for anything approaching AGI.</p>
  <p>AI Threat:</p>
  <p>From the standpoint of neoluddism, I'm skeptical that &quot;infinitely&quot; self-optimizing superintelligence is the greatest threat.  Evolution, and economic theory, suggest that the greater danger would come from not what is most expensive, complex, &amp; capable, but that which is &quot;<a href="https://en.wikipedia.org/wiki/Satisficing" rel="nofollow noreferrer">just good enough</a>&quot;, cheaper and more easily replicable.  This is because decision time becomes a critical factor in a competitive environment, as does replication time.</p>
  <p>I always bring up <a href="https://en.wikipedia.org/wiki/Gray_goo" rel="nofollow noreferrer">grey goo</a> because it only has to be good at 1 thing to eat everything else!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T01:19:25.487Z
Answers.LastActivityDate: 2020-08-19T01:19:25.487Z
Answers.UserDisplayName: DukeZhou
Body: <p>From Wikipedia:</p>
  <blockquote>
  <p>According to the most popular version of the singularity hypothesis, called intelligence explosion, an upgradable intelligent agent will eventually enter a &quot;runaway reaction&quot; of self-improvement cycles, each new and more intelligent generation appearing more and more rapidly, causing an &quot;explosion&quot; in intelligence and resulting in a powerful superintelligence that qualitatively far surpasses all human intelligence.</p>
  </blockquote>
  <p>But what if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?</p>
  <p>From experience we know that problems tend to be harder to solve at every iteration, with diminishing returns. Take as an example the theory of gravitation. Newtonian physics is relatively easy to formulate and covers the majority of high level gravitation phenomenas.
  A more refined picture, like General Relativity, fill few holes in the theory with a huge increase in complexity. To describe black holes and primordial cosmology we need a theory of quantum gravity, which appears to require a further step in complexity.</p>
  <p>What &quot;saved&quot; us so far is the economic growth of our civilisation, which allowed more and more scientist to focus on solving the next problems. It's true that the first AGI will have the luxury of being duplicated, being a software base intelligence, but at the same time is likely that the first AGI will be extremely compute intensive. But even assuming that we (or maybe it would better to say, they) have the hardware to run <span class="math-container">$10^{2}$</span> instances, if the complexity of every substantial self-improvement grows say by <span class="math-container">$10^{d}$</span>x with <span class="math-container">$d=3$</span> while the improvement to the intelligence is only <span class="math-container">$10^{l}$</span>x with <span class="math-container">$l=1$</span>, the self-improvement cycle will quickly slow down.</p>
  <p>So is increasing software complexity the most likely bottleneck to the AI singularity? And what are likely values for <span class="math-container">$d$</span> and <span class="math-container">$l$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-02T21:25:57.063Z
Comments.Text: "It's true that the first AGI will have the luxury of duplicating himself", it's true according to whom? The first AGI could actually not be able to duplicate itself, if by duplicate you mean reproduce. In fact, there are animals that can't procreate, such as mules (if I remember correctly), but they are still intelligent. So, I suggest that you revise your post to remove these "highly dubious" statements or at least use hypothetical statements. Moreover, I suggest that you ask a question that will not lead to opinions. Maybe you can formulate your question in that way.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-02T22:00:52.833Z
Comments.Text: I meant duplicated, edited the question. Likely the AGI will be created by an organisation, which could easily run multiple instances of it. I agree that self duplication at the beginning could be challenging.
Comments.UserDisplayName: Rexcirus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-03T02:39:08.840Z
Comments.Text: I think it's very difficult, if not impossible, to answer your questions appropriately, because, unless there's some mathematical theory that allows us to make predictions about a future AGI or SI, and how they will evolve, we can just speculate. My knowledge of physics is limited to high school, but I think a plausible prediction is that recursive self-improvement, if possible, will be a lot slower than people think.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-17T23:55:38.450Z
Comments.Text: It's a very interesting question, and I definitely understand what you are getting at.  (Might be worthwhile to look at this not from the standpoint of AGI or superintelligence, but from the standpoint of a predictive function, like [Laplace's Demon](https://en.wikipedia.org/wiki/Laplace%27s_demon#Cantor_diagonalization).  One idea of the singularity would be the idea of ultimately being able to simulating the entire observable universe, and the question of whether that is computationally possible.)  Does the cost of computation eventually become too great?
Comments.UserDisplayName: DukeZhou
Tags: <agi><singularity><kolmogorov-complexity>
Title: Is increasing software complexity the most likely bottleneck to the AI singularity?
UserDisplayName: Rexcirus
ThreadText: # Is increasing software complexity the most likely bottleneck to the AI singularity?
  
  (Asked by: username_0 on 2020-08-02)
  
  From Wikipedia:
  
  > According to the most popular version of the singularity hypothesis, called intelligence explosion, an upgradable intelligent agent will eventually enter a "runaway reaction" of self-improvement cycles, each new and more intelligent generation appearing more and more rapidly, causing an "explosion" in intelligence and resulting in a powerful superintelligence that qualitatively far surpasses all human intelligence.
  
  But what if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?
  
  From experience we know that problems tend to be harder to solve at every iteration, with diminishing returns. Take as an example the theory of gravitation. Newtonian physics is relatively easy to formulate and covers the majority of high level gravitation phenomenas.
  A more refined picture, like General Relativity, fill few holes in the theory with a huge increase in complexity. To describe black holes and primordial cosmology we need a theory of quantum gravity, which appears to require a further step in complexity.
  
  What "saved" us so far is the economic growth of our civilisation, which allowed more and more scientist to focus on solving the next problems. It's true that the first AGI will have the luxury of being duplicated, being a software base intelligence, but at the same time is likely that the first AGI will be extremely compute intensive. But even assuming that we (or maybe it would better to say, they) have the hardware to run $10^{2}$ instances, if the complexity of every substantial self-improvement grows say by $10^{d}$x with $d=3$ while the improvement to the intelligence is only $10^{l}$x with $l=1$, the self-improvement cycle will quickly slow down.
  
  So is increasing software complexity the most likely bottleneck to the AI singularity? And what are likely values for $d$ and $l$?
  
  **Tags:** agi, singularity, kolmogorov-complexity
  > **Comments:**
  > **username_1**: "It's true that the first AGI will have the luxury of duplicating himself", it's true according to whom? The first AGI could actually not be able to duplicate itself, if by duplicate you mean reproduce. In fact, there are animals that can't procreate, such as mules (if I remember correctly), but they are still intelligent. So, I suggest that you revise your post to remove these "highly dubious" statements or at least use hypothetical statements. Moreover, I suggest that you ask a question that will not lead to opinions. Maybe you can formulate your question in that way.  
  > **username_0**: I meant duplicated, edited the question. Likely the AGI will be created by an organisation, which could easily run multiple instances of it. I agree that self duplication at the beginning could be challenging.  
  > **username_1**: I think it's very difficult, if not impossible, to answer your questions appropriately, because, unless there's some mathematical theory that allows us to make predictions about a future AGI or SI, and how they will evolve, we can just speculate. My knowledge of physics is limited to high school, but I think a plausible prediction is that recursive self-improvement, if possible, will be a lot slower than people think.  
  > **username_2**: It's a very interesting question, and I definitely understand what you are getting at.  (Might be worthwhile to look at this not from the standpoint of AGI or superintelligence, but from the standpoint of a predictive function, like [Laplace's Demon](https://en.wikipedia.org/wiki/Laplace%27s_demon#Cantor_diagonalization).  One idea of the singularity would be the idea of ultimately being able to simulating the entire observable universe, and the question of whether that is computationally possible.)  Does the cost of computation eventually become too great?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-19. Score: 1
  
  > What if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?
  
  If this turns out to be the case, then the complexity would indeed be the bottleneck.
  
  There's a formal term for this: [kolmogrov complexity or "k-complexity"](https://en.wikipedia.org/wiki/Kolmogorov_complexity). (The best formal definition of intelligence I've come across is Hutter & Legg's, which is broken down in the article "[On the definition of intelligence](https://nbro.github.io/blogging/2020/05/20/on-the-definition-of-intelligence/)" by our own nbro. It will shed shed some light on the formal structure of the question at hand.
  
  * Any attempt to provide values to your d & l would speculative, since we're still so far from AGI. The question would be what is the effect of any given (d,l)
  
  The subject can definitely be treated formally in a mathematical sense the and values analyzed. (There may even been some formulae out there, although, they are likely to be more complex, per the Hutter&Legg.)
  
  Sorry I can't give you a more formal answer--it's a good question!
  
    
  
  Notes:
  
  Another bottleneck would be the speed of light, essentially, "bits and bytes across a pipeline". This would be a factor in both self-replication and networked processing and memory. You can expand bandwidth/throughput, but not infinitely.
  
  This relates to space complexity (how much volume/memory does the algorithm require?) in addition to the k-complexity, which itself is likely to be non-trivial for anything approaching AGI.
  
  AI Threat:
  
  From the standpoint of neoluddism, I'm skeptical that "infinitely" self-optimizing superintelligence is the greatest threat. Evolution, and economic theory, suggest that the greater danger would come from not what is most expensive, complex, & capable, but that which is "[just good enough](https://en.wikipedia.org/wiki/Satisficing)", cheaper and more easily replicable. This is because decision time becomes a critical factor in a competitive environment, as does replication time.
  
  I always bring up [grey goo](https://en.wikipedia.org/wiki/Gray_goo) because it only has to be good at 1 thing to eat everything else!
  
  
  ---
  
  