Body: <p>I’m experimenting with reinforcement learning for a 2D pixel plotting task, and am running into an issue that (I think) has to do with the big action space. It goes like this:</p>
  
  <p>The Agent gets two vector inputs each step.
  Each describes an (n x n) 2d matrix composed of zeros and ones. </p>
  
  <p>One is the (n x n) target matrix, containing a certain shape of zeros
  The other is an (n x n) state matrix, containing another shape</p>
  
  <p>Every step, I want my agent to pick an (x, y) coordinate:
  x (picks one of n)
  y (picks one of n)</p>
  
  <p>This will turn a zero into one, or one into zero.</p>
  
  <p>every step, if correct, I give a small reward, and it’ll get punished when incorrect.</p>
  
  <p>I’m training the agent (a network with 3 layers with 256 hidden units) with PPO, and curiosity in the loss, and for a 12 x 12 matrix it works quite well, not 100% but okay. (see image). Note that the agent doesn't get enough steps here to fully delete the initial shape when the target shape is empty, that's why it doesn't fully make it. Takes about 800K steps to converge though.
   <a href="https://i.sstatic.net/Y9XFx.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Y9XFx.gif" alt="example"></a></p>
  
  <p>But the agent starts struggling in local minima when I increase beyond 32 x 32.</p>
  
  <p>This one is at 32 x 32:</p>
  
  <p><a href="https://i.sstatic.net/gB1sX.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gB1sX.gif" alt="example at 32px"></a></p>
  
  <p>Is this even scalable to bigger matrices even? I was hoping to go 3D eventually, by reaching 100x100x100 .</p>
  
  <p>I do realize that i have a huge input and action space when working with such a grid. 
  Is something like that even possible with an RL paradigm? I’ve tried increasing the network size, and decreasing learning rate, but I’m not satisfied. Any ideas or alternative approaches to plot pixels like this?</p>
  
  <p>Any input is very much appreciated!
  Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-23T19:21:02.930Z
Comments.Text: [this](https://arxiv.org/pdf/1512.07679.pdf) paper might be relevant for you
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-23T19:56:52.433Z
Comments.Text: seems very relevant indeed. Will have a look!
Comments.UserDisplayName: SumakuTension
Tags: <machine-learning><reinforcement-learning>
Title: Reinforcement learning possible with big action space?
UserDisplayName: SumakuTension
ThreadText: # Reinforcement learning possible with big action space?
  
  (Asked by: username_0 on 2019-12-23)
  
  I’m experimenting with reinforcement learning for a 2D pixel plotting task, and am running into an issue that (I think) has to do with the big action space. It goes like this:
  
  The Agent gets two vector inputs each step.
  Each describes an (n x n) 2d matrix composed of zeros and ones.
  
  One is the (n x n) target matrix, containing a certain shape of zeros
  The other is an (n x n) state matrix, containing another shape
  
  Every step, I want my agent to pick an (x, y) coordinate:
  x (picks one of n)
  y (picks one of n)
  
  This will turn a zero into one, or one into zero.
  
  every step, if correct, I give a small reward, and it’ll get punished when incorrect.
  
  I’m training the agent (a network with 3 layers with 256 hidden units) with PPO, and curiosity in the loss, and for a 12 x 12 matrix it works quite well, not 100% but okay. (see image). Note that the agent doesn't get enough steps here to fully delete the initial shape when the target shape is empty, that's why it doesn't fully make it. Takes about 800K steps to converge though.
  [![example](https://i.sstatic.net/Y9XFx.gif)](https://i.sstatic.net/Y9XFx.gif)
  
  But the agent starts struggling in local minima when I increase beyond 32 x 32.
  
  This one is at 32 x 32:
  
  [![example at 32px](https://i.sstatic.net/gB1sX.gif)](https://i.sstatic.net/gB1sX.gif)
  
  Is this even scalable to bigger matrices even? I was hoping to go 3D eventually, by reaching 100x100x100 .
  
  I do realize that i have a huge input and action space when working with such a grid.
  Is something like that even possible with an RL paradigm? I’ve tried increasing the network size, and decreasing learning rate, but I’m not satisfied. Any ideas or alternative approaches to plot pixels like this?
  
  Any input is very much appreciated!
  Thanks!
  
  **Tags:** machine-learning, reinforcement-learning
  
  > **Comments:**
  > **username_1**: [this](https://arxiv.org/pdf/1512.07679.pdf) paper might be relevant for you  
  > **username_0**: seems very relevant indeed. Will have a look!  
  
  ---
  
  ## 0 answers
  
  