Answers.Body: <p>Thanks to the power of generalization of neural networks... the whole success of those models is their ability to be extremely effective in generalizing in high dimensions</p>
  <p>So, during training, the network is not learning the usual <span class="math-container">$|S|\times|A|$</span> Q-learning table, instead it's learning a compressed version of it, considering similarity between states</p>
  <p>Think about snake but with walls... instead of learning that in that position there is a wall, it learns that if it sees the head near a wall, it has to go in a direction that allows it not to smash into that (generalize)</p>
  <p>Obviously, this generalization, is only achieved if it's thought to the network, thus if during training, you also train it in different environments</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-08T14:33:54.957Z
Answers.Text: Thank you Alberto. I have not realized, and is a crucial aspect, that the agent (neural network) knows the different environment structures because it is encoded in the state vector. Knowing that, each state has one, and only one optimal action, and so is not so surprising that the network learns to act optimally. One could also do it as a look up table in the case of this example, because the number of different states is not so large. You are right that the network should be trained on the environment "random" mode. Else, it should not learn to cope with this situation.
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-08T14:51:09.383Z
Answers.Text: In the Zai&Brown example, the grid is 4 by 4. As explained in page 84, "The number  of possible game configurations (the size of the state-space) is approximately 16*15*14*13 =43680, since there are 16 possible positions the agent can be in on a 4*4 grid, and then 15 possible configurations for the wall, since tha agent and wall can´t be overlapping in space, etc.)". For a 5*5 grid we should have 25*24*23*22 = 303600  different configurations. No so much.
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-08T14:55:40.537Z
Answers.Text: @HermesMorales you know what? I'm so used to look at functions, that I forgot how to do multiplications... however, we agree that the more the walls, the more the states, and If im not mistaken again, with more than 5 walls, get's pretty large, so you have to rely on generalization
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T20:34:40.963Z
Answers.LastActivityDate: 2023-08-07T20:34:40.963Z
Answers.UserDisplayName: Alberto
Body: <p>I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that the goal, obstacles, and player will be distributed differently each time.</p>
  <p>When testing the learning on the randomly initialized grid, the simpler network - a vanilla one - is able to win roughly 50% of the time. When an 'experience replay' mechanism is added, it attains more than 90% success. My question is how the network is able to achieve these results, given that - for a given state - the optimal action will be different every time the environment is initialized.</p>
  <p>You can find the full code in</p>
  <p><a href="https://github.com/DeepReinforcementLearning/DeepReinforcementLearningInAction/tree/master/Chapter%203" rel="nofollow noreferrer">DRL in action Chap 3 github page</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-08T14:36:47.797Z
Comments.Text: I was wrong. It is not true that for a given state the optimal action will be different every time the environment is initialized.
Comments.UserDisplayName: Hermes Morales
Tags: <deep-rl><dqn>
Title: How does DeepQ learn for different environments?
UserDisplayName: Hermes Morales
ThreadText: # How does DeepQ learn for different environments?
  
  (Asked by: username_0 on 2023-08-07)
  
  I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that the goal, obstacles, and player will be distributed differently each time.
  
  When testing the learning on the randomly initialized grid, the simpler network - a vanilla one - is able to win roughly 50% of the time. When an 'experience replay' mechanism is added, it attains more than 90% success. My question is how the network is able to achieve these results, given that - for a given state - the optimal action will be different every time the environment is initialized.
  
  You can find the full code in
  
  [DRL in action Chap 3 github page](https://github.com/DeepReinforcementLearning/DeepReinforcementLearningInAction/tree/master/Chapter%203)
  
  **Tags:** deep-rl, dqn
  
  > **Comments:**
  > **username_0**: I was wrong. It is not true that for a given state the optimal action will be different every time the environment is initialized.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-07. Score: 1
  
  Thanks to the power of generalization of neural networks... the whole success of those models is their ability to be extremely effective in generalizing in high dimensions
  
  So, during training, the network is not learning the usual $|S|\times|A|$ Q-learning table, instead it's learning a compressed version of it, considering similarity between states
  
  Think about snake but with walls... instead of learning that in that position there is a wall, it learns that if it sees the head near a wall, it has to go in a direction that allows it not to smash into that (generalize)
  
  Obviously, this generalization, is only achieved if it's thought to the network, thus if during training, you also train it in different environments
  
  > **Comments:**
  > **username_0**: Thank you username_1. I have not realized, and is a crucial aspect, that the agent (neural network) knows the different environment structures because it is encoded in the state vector. Knowing that, each state has one, and only one optimal action, and so is not so surprising that the network learns to act optimally. One could also do it as a look up table in the case of this example, because the number of different states is not so large. You are right that the network should be trained on the environment "random" mode. Else, it should not learn to cope with this situation.  
  > **username_0**: In the Zai&Brown example, the grid is 4 by 4. As explained in page 84, "The number  of possible game configurations (the size of the state-space) is approximately 16*15*14*13 =43680, since there are 16 possible positions the agent can be in on a 4*4 grid, and then 15 possible configurations for the wall, since tha agent and wall can´t be overlapping in space, etc.)". For a 5*5 grid we should have 25*24*23*22 = 303600  different configurations. No so much.  
  > **username_1**: @username_0 you know what? I'm so used to look at functions, that I forgot how to do multiplications... however, we agree that the more the walls, the more the states, and If im not mistaken again, with more than 5 walls, get's pretty large, so you have to rely on generalization  
  
  ---
  
  