Body: <p>There have been instances of agents using edge cases like bugs in <a href="https://www.youtube.com/watch?v=Lu56xVlZ40M" rel="nofollow noreferrer">physics engines</a>, <a href="https://www.gwern.net/Tanks#alternative-examples" rel="nofollow noreferrer">repetitive behavior in games</a> or word repetition in text prediction to cheat their reward function. However, these agents are arguably still contained, as while they explore the extremes of the state space of the simulation they don't expand their action space beyond what is possible in the simulation.</p>
  <p>The <a href="https://www.youtube.com/watch?v=p5T81yHkHtI" rel="nofollow noreferrer">Pokémon Yellow Total Control Hack</a> shows that in some systems, it is possible to gain full control of a computer by exploiting bugs in the hardware or software (here: memory corruption), enabling the agent to even <em>completely</em> reprogram the system 'from within', just using the normal inputs.</p>
  <p>Do you know of similar extreme examples where an AI agent went far beyond what was intended with the simulation environment and expanded its action space?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-02T18:29:17.150Z
Comments.Text: This kind of scenario is still solidly in the realms of science fiction. The current crop of optimising and statistical pattern AIs are not "minds" in any common sense understanding of the word.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-02T22:20:24.100Z
Comments.Text: "There have been instances of agents using edge cases like...", can you please provide a reference that supports all your examples?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T07:20:20.163Z
Comments.Text: @nbro: The first link is to a short video which does demonstrate a game physics exploit being used to gain reward for a task. I think someone has collated a few similar examples on a page somewhere, I'll see if I can find it
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T07:25:19.383Z
Comments.Text: @nbro: Here's a good list, and it might be the foundation of an answer: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml (but as per my comment, the idea of an agent somehow "evolving" or transcending its state/action space to attack the hardware or redefine itself is still far out of reach).
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-04T22:22:37.477Z
Comments.Text: @NeilSlater I had come across that list in the past and I had completely forgotten about it when I wrote my comment above. Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T20:47:52.930Z
Comments.Text: This is actually a very interesting question, but it might be a good idea to clarify what you mean by this "agent went far beyond what was intended with the simulation environment". In my view, the question is a bit vague, and the examples that Neil provide could actually be what you're looking for or not. So, if they are not, maybe you can edit your post to explain why and refer to them.
Comments.UserDisplayName: nbro
Tags: <intelligent-agent><ai-safety><ai-box><reward-hacking>
Title: Has there been an instance of an AI agent breaking out of its sandbox?
UserDisplayName: 2080
ThreadText: # Has there been an instance of an AI agent breaking out of its sandbox?
  
  (Asked by: username_0 on 2022-07-02)
  
  There have been instances of agents using edge cases like bugs in [physics engines](https://www.youtube.com/watch?v=Lu56xVlZ40M), [repetitive behavior in games](https://www.gwern.net/Tanks#alternative-examples) or word repetition in text prediction to cheat their reward function. However, these agents are arguably still contained, as while they explore the extremes of the state space of the simulation they don't expand their action space beyond what is possible in the simulation.
  
  The [Pokémon Yellow Total Control Hack](https://www.youtube.com/watch?v=p5T81yHkHtI) shows that in some systems, it is possible to gain full control of a computer by exploiting bugs in the hardware or software (here: memory corruption), enabling the agent to even *completely* reprogram the system 'from within', just using the normal inputs.
  
  Do you know of similar extreme examples where an AI agent went far beyond what was intended with the simulation environment and expanded its action space?
  
  **Tags:** intelligent-agent, ai-safety, ai-box, reward-hacking
  
  > **Comments:**
  > **username_1**: This kind of scenario is still solidly in the realms of science fiction. The current crop of optimising and statistical pattern AIs are not "minds" in any common sense understanding of the word.  
  > **username_2**: "There have been instances of agents using edge cases like...", can you please provide a reference that supports all your examples?  
  > **username_1**: @username_2: The first link is to a short video which does demonstrate a game physics exploit being used to gain reward for a task. I think someone has collated a few similar examples on a page somewhere, I'll see if I can find it  
  > **username_1**: @username_2: Here's a good list, and it might be the foundation of an answer: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml (but as per my comment, the idea of an agent somehow "evolving" or transcending its state/action space to attack the hardware or redefine itself is still far out of reach).  
  > **username_2**: @username_1 I had come across that list in the past and I had completely forgotten about it when I wrote my comment above. Thanks.  
  > **username_2**: This is actually a very interesting question, but it might be a good idea to clarify what you mean by this "agent went far beyond what was intended with the simulation environment". In my view, the question is a bit vague, and the examples that username_1 provide could actually be what you're looking for or not. So, if they are not, maybe you can edit your post to explain why and refer to them.  
  
  ---
  
  ## 0 answers
  
  