Body: <p>What would happen when an artificial general intelligence can improve itself over a long time, with limited resources?</p>
  
  <p>The assumption is that it has a large but finite amount of computing power, and can not escape that limit to find resources. Let's assume the limit to be on the order 100 human brains or so - the limits are not important, only that it is limited.</p>
  
  <p>Now, we let it run to self-improve as much as it can with the given resources, until it goes to some stable or periodic state.</p>
  
  <p>What it can do is at least one of the following:  </p>
  
  <ul>
  <li>converge to some state</li>
  <li>oscillate between two or more states</li>
  <li>go into a chaotic state</li>
  <li>stop doing anything</li>
  <li>disable itself in some other way than just stop doing anything</li>
  </ul>
  
  <p>There are certainly more complex behaviors possible:</p>
  
  <ul>
  <li>become maximally happy, and then periodically redefine what happy means</li>
  <li>hacking its reward function in some other way</li>
  </ul>
  
  <p>I would expect it to converges to a single state - but that seems naive.</p>
  
  <p>Are there any ideas on how it would end up?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T20:10:00.057Z
Comments.Text: To be clear, you're assuming that you have an AGI. Right? So, this question assumes that there's an AGI already that is able to self-improve. Right? You talk about states, but you don't define states, so this is a bit dangerous. 
  
  I suggest you simplify as much as possible your post and question and try to make it clear what you want to ask and define the terms you are using. This is clearly a philosophical, question, but not only.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T20:11:31.207Z
Comments.Text: Also, you talk about "finite resources" but also about "infinite time". Isn't time also a resource? So, you should probably clearly define what you mean by "resource". I am trying to make this question a very good question. I think that it's already interesting, but, hopefully, we can make it more scientifically answerable.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T20:36:36.087Z
Comments.Text: @nbro Yes, the term "resource" is too general there. In terms of finite ressources, but infinite time: I do not want to limit how long it takes, I' want to wait until it gets stable or periodic, basically. "Wait until nothing new happens". Yes, that are all good points, thanks!
Comments.UserDisplayName: Volker Siegel
Tags: <philosophy><agi><superintelligence><singularity>
Title: What does a self-improving artificial general intelligence with finite resources and infinite time do?
UserDisplayName: Volker Siegel
ThreadText: # What does a self-improving artificial general intelligence with finite resources and infinite time do?
  
  (Asked by: username_0 on 2020-05-05)
  
  What would happen when an artificial general intelligence can improve itself over a long time, with limited resources?
  
  The assumption is that it has a large but finite amount of computing power, and can not escape that limit to find resources. Let's assume the limit to be on the order 100 human brains or so - the limits are not important, only that it is limited.
  
  Now, we let it run to self-improve as much as it can with the given resources, until it goes to some stable or periodic state.
  
  What it can do is at least one of the following:
  
  * converge to some state
  * oscillate between two or more states
  * go into a chaotic state
  * stop doing anything
  * disable itself in some other way than just stop doing anything
  
  There are certainly more complex behaviors possible:
  
  * become maximally happy, and then periodically redefine what happy means
  * hacking its reward function in some other way
  
  I would expect it to converges to a single state - but that seems naive.
  
  Are there any ideas on how it would end up?
  
  **Tags:** philosophy, agi, superintelligence, singularity
  
  > **Comments:**
  > **username_1**: To be clear, you're assuming that you have an AGI. Right? So, this question assumes that there's an AGI already that is able to self-improve. Right? You talk about states, but you don't define states, so this is a bit dangerous.   I suggest you simplify as much as possible your post and question and try to make it clear what you want to ask and define the terms you are using. This is clearly a philosophical, question, but not only.  
  > **username_1**: Also, you talk about "finite resources" but also about "infinite time". Isn't time also a resource? So, you should probably clearly define what you mean by "resource". I am trying to make this question a very good question. I think that it's already interesting, but, hopefully, we can make it more scientifically answerable.  
  > **username_0**: @username_1 Yes, the term "resource" is too general there. In terms of finite ressources, but infinite time: I do not want to limit how long it takes, I' want to wait until it gets stable or periodic, basically. "Wait until nothing new happens". Yes, that are all good points, thanks!  
  
  ---
  
  ## 0 answers
  
  