<p>I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
One main reason for this choice: when exploiting the learnt function (i.e. policy is <span class="math-container">$\arg \max_a Q(s,a)$</span>), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.</p>

CC BY-SA 4.0
2020-12-19T09:29:18.640Z
2020-12-19T09:29:18.640Z
rhdxor
<p>I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.</p>
<p>Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?</p>
<p>Here are the two options I can think of.</p>
<ol>
<li><p>The input of the neural network is <span class="math-container">$(s_i, a_i)$</span> and the output is <span class="math-container">$Q(s_i,a_i)$</span></p>
</li>
<li><p>The input is <span class="math-container">$(s_i)$</span> and the output is a vector <span class="math-container">$[Q(s_i,a_1), Q(s_i,a_2), \dots, Q(s_i,a_N)]$</span></p>
</li>
</ol>
<p>Is there any other alternative architecture?</p>
<p>Also, how to reason about which model would be logically better?</p>

CC BY-SA 4.0
2020-12-19T13:27:36.717Z
Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) paper. In any case, this question seems to be a duplicate of [this one](https://ai.stackexchange.com/q/21669/2444).
nbro
<neural-networks><reinforcement-learning><q-learning><dqn><deep-rl>
How to build a Neural Network to approximate the Q-function?
Exploring
# How to build a Neural Network to approximate the Q-function?

(Asked by: username_0 on 2020-12-19)

I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.

Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?

Here are the two options I can think of.

1. The input of the neural network is $(s\_i, a\_i)$ and the output is $Q(s\_i,a\_i)$
2. The input is $(s\_i)$ and the output is a vector $[Q(s\_i,a\_1), Q(s\_i,a\_2), \dots, Q(s\_i,a\_N)]$

Is there any other alternative architecture?

Also, how to reason about which model would be logically better?

**Tags:** neural-networks, reinforcement-learning, q-learning, dqn, deep-rl

> **Comments:**
> **username_1**: Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) paper. In any case, this question seems to be a duplicate of [this one](https://ai.stackexchange.com/q/21669/2444).  

---

## 1 answers

### Answer by username_2 on 2020-12-19. Score: 2

I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
One main reason for this choice: when exploiting the learnt function (i.e. policy is $\arg \max\_a Q(s,a)$), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.


---

