Answers.Body: <p>I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across &quot;similar&quot; states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T17:58:57.057Z
Answers.Text: But surely for large state space problems, the same method would apply? There's an example in a book called "Deep Reinforcement Learning in Action" that has an example of a NN being trained to learn to play Gridworld, which is what I'm basing my question off. So I suppose what I'm asking is: do we use NNs because they're much more efficient at handling the task?
Answers.UserDisplayName: THAT_AI_GUY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T18:37:25.527Z
Answers.Text: In general, as stated above, nowadays, NNs are the preferred choice (over traditional, tabular approaches to RL) because of the reasons mentioned in the answer above. But if you apply NN to very simple problems, I see two main reasons: 1) It's super convenient to use them because super-efficient, scalable implementations are available for free on the internet; 2) often applying a NN-based  RL-algorithm to a simple task can be seen as some kind of proof of concept (and for illustration). If you can show that it works nicely on small problems, then it will theoretically also work on larger ones.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T18:43:22.693Z
Answers.Text: Besides that, even for games that look super simple (superficially), the number of possible states can grow tremendously large pretty quickly. So, even if you could (still) fit all those states in a table, the required training time could be immense. Just because there is just no generalization in the training outcome when training a tabular RL agent.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T17:45:52.003Z
Answers.LastActivityDate: 2020-07-16T17:45:52.003Z
Answers.UserDisplayName: harwiltz
Body: <p>I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.</p>
  <p>Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.</p>
  <p>In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.</p>
  <p>So:</p>
  <p>(a) Q-learning update rule-&gt; outputs target Q-values</p>
  <p>(b) NN -&gt; outputs Q values</p>
  <p>MSE to compute the loss between (a) and (b)</p>
  <p>So, given we already know what the target Q-value is from a, why do we need to train a NN?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-16T21:20:00.257Z
Comments.Text: Can you please link us to the algorithm that you have seen where the target Q values are not produced by a NN but an NN is still used in the algorithm?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-16T22:22:37.033Z
Comments.Text: Hello, what I mean is the target is produced by a Q-learning update rule, and the NN is what is trained to mimic the behaviour of the Q-learning update rule. The example is from a book called "Deep RL in Action" and the example can be found on pages 54-68.
Comments.UserDisplayName: THAT_AI_GUY
Tags: <neural-networks><reinforcement-learning><deep-learning><training><q-learning>
Title: What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?
UserDisplayName: THAT_AI_GUY
ThreadText: # What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?
  
  (Asked by: username_0 on 2020-07-16)
  
  I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.
  
  Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.
  
  In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.
  
  So:
  
  (a) Q-learning update rule-> outputs target Q-values
  
  (b) NN -> outputs Q values
  
  MSE to compute the loss between (a) and (b)
  
  So, given we already know what the target Q-value is from a, why do we need to train a NN?
  
  **Tags:** neural-networks, reinforcement-learning, deep-learning, training, q-learning
  > **Comments:**
  > **username_1**: Can you please link us to the algorithm that you have seen where the target Q values are not produced by a NN but an NN is still used in the algorithm?  
  > **username_0**: Hello, what I mean is the target is produced by a Q-learning update rule, and the NN is what is trained to mimic the behaviour of the Q-learning update rule. The example is from a book called "Deep RL in Action" and the example can be found on pages 54-68.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-16. Score: 3
  
  I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across "similar" states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.
  
  > **Comments:**
  > **username_0**: But surely for large state space problems, the same method would apply? There's an example in a book called "Deep Reinforcement Learning in Action" that has an example of a NN being trained to learn to play Gridworld, which is what I'm basing my question off. So I suppose what I'm asking is: do we use NNs because they're much more efficient at handling the task?  
  > **username_3**: In general, as stated above, nowadays, NNs are the preferred choice (over traditional, tabular approaches to RL) because of the reasons mentioned in the answer above. But if you apply NN to very simple problems, I see two main reasons: 1) It's super convenient to use them because super-efficient, scalable implementations are available for free on the internet; 2) often applying a NN-based  RL-algorithm to a simple task can be seen as some kind of proof of concept (and for illustration). If you can show that it works nicely on small problems, then it will theoretically also work on larger ones.  
  > **username_3**: Besides that, even for games that look super simple (superficially), the number of possible states can grow tremendously large pretty quickly. So, even if you could (still) fit all those states in a table, the required training time could be immense. Just because there is just no generalization in the training outcome when training a tabular RL agent.  
  
  ---
  
  