Answers.Body: <p>I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
  One main reason for this choice: when exploiting the learnt function (i.e. policy is <span class="math-container">$\arg \max_a Q(s,a)$</span>), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-19T09:29:18.640Z
Answers.LastActivityDate: 2020-12-19T09:29:18.640Z
Answers.UserDisplayName: rhdxor
Body: <p>I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.</p>
  <p>Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?</p>
  <p>Here are the two options I can think of.</p>
  <ol>
  <li><p>The input of the neural network is <span class="math-container">$(s_i, a_i)$</span> and the output is <span class="math-container">$Q(s_i,a_i)$</span></p>
  </li>
  <li><p>The input is <span class="math-container">$(s_i)$</span> and the output is a vector <span class="math-container">$[Q(s_i,a_1), Q(s_i,a_2), \dots, Q(s_i,a_N)]$</span></p>
  </li>
  </ol>
  <p>Is there any other alternative architecture?</p>
  <p>Also, how to reason about which model would be logically better?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-19T13:27:36.717Z
Comments.Text: Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) paper. In any case, this question seems to be a duplicate of [this one](https://ai.stackexchange.com/q/21669/2444).
Comments.UserDisplayName: nbro
Tags: <neural-networks><reinforcement-learning><q-learning><dqn><deep-rl>
Title: How to build a Neural Network to approximate the Q-function?
UserDisplayName: Exploring
ThreadText: # How to build a Neural Network to approximate the Q-function?
  
  (Asked by: username_0 on 2020-12-19)
  
  I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.
  
  Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?
  
  Here are the two options I can think of.
  
  1. The input of the neural network is $(s\_i, a\_i)$ and the output is $Q(s\_i,a\_i)$
  2. The input is $(s\_i)$ and the output is a vector $[Q(s\_i,a\_1), Q(s\_i,a\_2), \dots, Q(s\_i,a\_N)]$
  
  Is there any other alternative architecture?
  
  Also, how to reason about which model would be logically better?
  
  **Tags:** neural-networks, reinforcement-learning, q-learning, dqn, deep-rl
  
  > **Comments:**
  > **username_1**: Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) paper. In any case, this question seems to be a duplicate of [this one](https://ai.stackexchange.com/q/21669/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-19. Score: 2
  
  I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
  One main reason for this choice: when exploiting the learnt function (i.e. policy is $\arg \max\_a Q(s,a)$), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.
  
  
  ---
  
  