Answers.Body: <blockquote>
  <p>Would it be cheaper to have <span class="math-container">$N$</span> neural networks with a single real-valued output, one for each of the <span class="math-container">$N$</span> actions?</p>
  </blockquote>
  <p>I think the &quot;<a href="https://en.wikipedia.org/wiki/No_free_lunch_theorem" rel="nofollow noreferrer">No Free Lunch</a>&quot; theorem applies here, or something like it.</p>
  <p>Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:</p>
  <p><em>The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.</em></p>
  <p>This is only an educated guess.</p>
  <p>As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-28T19:29:26.767Z
Answers.LastActivityDate: 2021-04-28T21:19:08.387Z
Answers.LastEditDate: 2021-04-28T21:19:08.387Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of <span class="math-container">$Q(s, a)$</span> predicted for all the possible <span class="math-container">$N$</span> actions.</p>
  <p>Would it be cheaper to have <span class="math-container">$N$</span> neural networks with a single real-valued output, one for each of the <span class="math-container">$N$</span> actions?</p>
  <p>With cheaper I mean cheaper in terms of the time complexity of a single training step of the network.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-28T10:04:35.243Z
Comments.Text: Hello. I tried to clarify your question. Make sure that I didn't change the meaning of the post. Please, could you also clarify what you mean by "cheaper"? Cheaper in terms of what?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-28T13:21:35.490Z
Comments.Text: Hi, thank you. The meaning is the same. With "cheaper" I mean cheaper in terms of operations required.
Comments.UserDisplayName: Antonio Sannia
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-29T10:27:25.463Z
Comments.Text: Please, edit your post to clarify what you mean by "cheap". In terms of operations, do you mean the time complexity of one iteration of training DQN as described in the original paper?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-29T20:18:22.357Z
Comments.Text: Yes, I mean that. I have edited.
Comments.UserDisplayName: Antonio Sannia
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-29T22:27:33.420Z
Comments.Text: But, if that's what you mean, why did you accept the answer below? It doesn't really answer your question (which was not specific enough indeed). You should discuss your edited version with the author of that answer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T06:48:23.357Z
Comments.Text: @nbro: Rating learning on time complexity alone is not useful. You can easily trade accuracy or efficiency of learning for time complexity, up to the extremes of not learning anything very fast per iteration. So there needs to be some further constraints. However, it is very hard to phrase such trade-offs in a question like this. I think my answer (or some variant of it) is indeed what the OP is looking for - validation that the idea has some merit, with caveats on which scenarios it could be useful in.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T06:51:11.247Z
Comments.Text: The only improvements I can see is that both question and answer could be extended to cover more technical details on the applied scenarios and degree to which the idea has merit. However, that is hard to phrase correctly. Definitely time complexity per step alone, without specifying the variables of interest, is not a useful metric.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T09:27:36.113Z
Comments.Text: @NeilSlater You're right that the time complexity of these stochastic algorithms may not be the first metric that one may be interested in, but I don't think it's completely useless (in particular, in the cases the proposed algorithms converge to the same solutions, which doesn't seem will be the case here). Anyway, my point was that the question wasn't clear/specific enough, so I think using the word "cheap" here is very vague. Maybe the OP can confirm that he was actually not interested in the time complexity. In that case, you may want to suggest another word, or rephrase the question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><deep-rl><dqn>
Title: In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
UserDisplayName: Antonio Sannia
ThreadText: # In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
  
  (Asked by: username_0 on 2021-04-27)
  
  In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted for all the possible $N$ actions.
  
  Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
  
  With cheaper I mean cheaper in terms of the time complexity of a single training step of the network.
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, dqn
  > **Comments:**
  > **username_1**: Hello. I tried to clarify your question. Make sure that I didn't change the meaning of the post. Please, could you also clarify what you mean by "cheaper"? Cheaper in terms of what?  
  > **username_0**: Hi, thank you. The meaning is the same. With "cheaper" I mean cheaper in terms of operations required.  
  > **username_1**: Please, edit your post to clarify what you mean by "cheap". In terms of operations, do you mean the time complexity of one iteration of training DQN as described in the original paper?  
  > **username_0**: Yes, I mean that. I have edited.  
  > **username_1**: But, if that's what you mean, why did you accept the answer below? It doesn't really answer your question (which was not specific enough indeed). You should discuss your edited version with the author of that answer.  
  > **username_2**: @nbro: Rating learning on time complexity alone is not useful. You can easily trade accuracy or efficiency of learning for time complexity, up to the extremes of not learning anything very fast per iteration. So there needs to be some further constraints. However, it is very hard to phrase such trade-offs in a question like this. I think my answer (or some variant of it) is indeed what the OP is looking for - validation that the idea has some merit, with caveats on which scenarios it could be useful in.  
  > **username_2**: The only improvements I can see is that both question and answer could be extended to cover more technical details on the applied scenarios and degree to which the idea has merit. However, that is hard to phrase correctly. Definitely time complexity per step alone, without specifying the variables of interest, is not a useful metric.  
  > **username_1**: @NeilSlater You're right that the time complexity of these stochastic algorithms may not be the first metric that one may be interested in, but I don't think it's completely useless (in particular, in the cases the proposed algorithms converge to the same solutions, which doesn't seem will be the case here). Anyway, my point was that the question wasn't clear/specific enough, so I think using the word "cheap" here is very vague. Maybe the OP can confirm that he was actually not interested in the time complexity. In that case, you may want to suggest another word, or rephrase the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-04-28. Score: 3 | Accepted answer
  
  > Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
  
  I think the "[No Free Lunch](https://en.wikipedia.org/wiki/No_free_lunch_theorem)" theorem applies here, or something like it.
  
  Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:
  
  *The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.*
  
  This is only an educated guess.
  
  As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.
  
  
  ---
  
  