Answers.Body: <p>You don't explain how you convert state to neural network features. Which makes me think that you are doing no feature scaling, using the state directly as input. This is a common mistake when implementing deep RL. The neural network can perform badly with inputs outside of range -2 to +2. <em>So one thing you can do is scale the state vector before using it as input to the neural network.</em></p>
  <p>Another thing that might be useful because you have a large number of actions is to look at double DQN, where you run two updating  estimators, using one to select the maximising action but the other to generate the TD target from it. This reduces the bias from taking the &quot;best&quot; action from lots of approximations, and then using the very likely overestimated action value from the same estimate.</p>
  <p>I can't rule out other sources of error here. Other possibilities include that your state is missing key data that impacts future reward. I'm not sure if that might be the case from your description.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T13:01:47.760Z
Answers.Text: used this to convert to 1d array
  
  ```def stateDictToArray(self, state):
          return np.concatenate([
                  state["yard"].reshape(-1)
                  ])  ```
Answers.UserDisplayName: Betnem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T13:10:16.430Z
Answers.Text: just add an image to my question.
Answers.UserDisplayName: Betnem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T13:52:45.110Z
Answers.Text: @Betnem If you are using values of 30 or over as inputs to the neural network, my first paragraph definitley applies. This will make the NN perform badly.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T14:04:19.140Z
Answers.Text: thanks a lot. 
  I will make a change in my state to 0 to represent void slots and another value to represent occupied slots... thanks!
Answers.UserDisplayName: Betnem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T14:30:12.583Z
Answers.Text: Didnt get any good results...
Answers.UserDisplayName: Betnem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T14:32:43.680Z
Answers.Text: @Betnem That's a shame. Sorry I am out of ideas what could be wrong. I think you need someone closer to your project to work with you, or just keep trying ideas. I hope the little bit I've been able to suggest turns out to be a useful step towards getting it working
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T12:11:02.610Z
Answers.LastActivityDate: 2023-10-04T12:11:02.610Z
Answers.UserDisplayName: Neil Slater
Body: <p>I created a custom env to simulate a sort of blocks using gym enviroments.</p>
  <p>My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack.</p>
  <p><code>yard: spaces.Box(low=-np.inf, high=np.inf, shape=(8, 8), dtype=float)</code></p>
  <p>The action space is discrete, being mapped to a dictionary that contains all POSSIBLE movements in this environment, which total 168, namely:</p>
  <pre><code>self.ACTION_LOOKUP = {0: &quot;(0,1,0) - top slab from stack 0 to top of stack 1&quot;,
  2: &quot;(0,1,1)&quot; - two
                                 1: &quot;(0,2, 2) - top three slabs from post 0 to top of stack 2&quot;}
  </code></pre>
  <p>My reset, in addition to starting the variables that are used throughout the environment, generates a &quot;random map&quot;, based on the seed.
  This map is the arrangement of blocks and location of which must be unlocked in the correct sequence.</p>
  <p>Once this is done, the step is designed for 3 main rules</p>
  <ol>
  <li>Do not allow an action that is not in space to be performed (this scenario only occurs when the rendering mode is like HUMAN)</li>
  <li>Do not allow an action to be performed starting from an empty stack or with fewer blocks than what is being requested.</li>
  <li>Do not allow a stack to receive more blocks than its capacity.</li>
  </ol>
  <p>In this case the action is not performed and the observation space remains the same.</p>
  <p>The reward function, which I have already implemented countless times... is currently designed to capture the general state of the environment and no longer the UNIQUE MOVEMENT (the results are not the same when compared in the evolution of reward_mean).
  In this case, it is a complex punctuation composition:</p>
  <ol>
  <li>If the above validations are checked (of invalid movements, return -1)</li>
  </ol>
  <p>Otherwise, I EXECUTE THE CHANGE IN THE STATE OF THE ENVIRONMENT, I calculate the reward (after the movement has been executed), according to the following composition:</p>
  <ol>
  <li>1 point for moving a correct pile (where it contains a block)</li>
  <li>Percentage 0 to 100, of how close I can complete unlocking the blocks - this calculation is based on proximity to the surface</li>
  <li>Additional points for partially unlocking a block</li>
  <li>Reducer (-1) for the current step, my intention is for the game to be completed in the least amount of steps possible.</li>
  </ol>
  <hr />
  <p><a href="https://i.sstatic.net/Fmltk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Fmltk.png" alt="Dict of my state + image rendered as human" /></a></p>
  <p><a href="https://i.sstatic.net/VC9bG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VC9bG.png" alt="Sad :( - Always decrease" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T10:57:51.160Z
Comments.Text: @NeilSlater thanks, I reviewed my question. Take a look if it makes sense now.
Comments.UserDisplayName: Betnem
Tags: <dqn><open-ai><gym>
Title: Help for my custom ENV with GYM, trying with DQN
UserDisplayName: Betnem
ThreadText: # Help for my custom ENV with GYM, trying with DQN
  
  (Asked by: username_0 on 2023-10-04)
  
  I created a custom env to simulate a sort of blocks using gym enviroments.
  My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack.
  yard: spaces.Box(low=-np.inf, high=np.inf, shape=(8, 8), dtype=float)
  The action space is discrete, being mapped to a dictionary that contains all POSSIBLE movements in this environment, which total 168, namely:
  self.ACTION_LOOKUP = {0: "(0,1,0) - top slab from stack 0 to top of stack 1",
  2: "(0,1,1)" - two
                                 1: "(0,2, 2) - top three slabs from post 0 to top of stack 2"}
  
  My reset, in addition to starting the variables that are used throughout the environment, generates a "random map", based on the seed.
  This map is the arrangement of blocks and location of which must be unlocked in the correct sequence.
  Once this is done, the step is designed for 3 main rules
  
  Do not allow an action that is not in space to be performed (this scenario only occurs when the rendering mode is like HUMAN)
  Do not allow an action to be performed starting from an empty stack or with fewer blocks than what is being requested.
  Do not allow a stack to receive more blocks than its capacity.
  
  In this case the action is not performed and the observation space remains the same.
  The reward function, which I have already implemented countless times... is currently designed to capture the general state of the environment and no longer the UNIQUE MOVEMENT (the results are not the same when compared in the evolution of reward_mean).
  In this case, it is a complex punctuation composition:
  
  If the above validations are checked (of invalid movements, return -1)
  
  Otherwise, I EXECUTE THE CHANGE IN THE STATE OF THE ENVIRONMENT, I calculate the reward (after the movement has been executed), according to the following composition:
  
  1 point for moving a correct pile (where it contains a block)
  Percentage 0 to 100, of how close I can complete unlocking the blocks - this calculation is based on proximity to the surface
  Additional points for partially unlocking a block
  Reducer (-1) for the current step, my intention is for the game to be completed in the least amount of steps possible.
  
  **Tags:** dqn, open-ai, gym
  > **Comments:**
  > **username_0**: @NeilSlater thanks, I reviewed my question. Take a look if it makes sense now.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-04. Score: 1
  
  You don't explain how you convert state to neural network features. Which makes me think that you are doing no feature scaling, using the state directly as input. This is a common mistake when implementing deep RL. The neural network can perform badly with inputs outside of range -2 to +2. *So one thing you can do is scale the state vector before using it as input to the neural network.*
  
  Another thing that might be useful because you have a large number of actions is to look at double DQN, where you run two updating estimators, using one to select the maximising action but the other to generate the TD target from it. This reduces the bias from taking the "best" action from lots of approximations, and then using the very likely overestimated action value from the same estimate.
  
  I can't rule out other sources of error here. Other possibilities include that your state is missing key data that impacts future reward. I'm not sure if that might be the case from your description.
  
  > **Comments:**
  > **username_0**: used this to convert to 1d array  ```def stateDictToArray(self, state):         return np.concatenate([                 state["yard"].reshape(-1)                 ])  ```  
  > **username_0**: just add an image to my question.  
  > **username_1**: @Betnem If you are using values of 30 or over as inputs to the neural network, my first paragraph definitley applies. This will make the NN perform badly.  
  > **username_0**: thanks a lot.  I will make a change in my state to 0 to represent void slots and another value to represent occupied slots... thanks!  
  > **username_0**: Didnt get any good results...  
  > **username_1**: @Betnem That's a shame. Sorry I am out of ideas what could be wrong. I think you need someone closer to your project to work with you, or just keep trying ideas. I hope the little bit I've been able to suggest turns out to be a useful step towards getting it working  
  
  ---
  
  