Answers.Body: <p>Predicting the correct amount of repetitions for an action sounds like a <strong>regression</strong> task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being:</p>
  <ul>
  <li>Having to choose a priori a finite max amount of actions n</li>
  <li>Turning the data into really sparse vectors, especially for large n.</li>
  </ul>
  <p>So a better choice in my opinion would be to treat this problem as a regression one, i.e. using a single output node trained on predicting continuous values, and round the predictions to convert real values to integers.</p>
  <p>Of course the task of choosing an action is a classification one, but it is possible to combine classification and regression into a single model, using <a href="https://ruder.io/multi-task/" rel="nofollow noreferrer">Multi Task Learning</a>, the idea being to train a model with one head (input layer) one body (some hidden layers) and 2 tails (output layers), one trained on classification and one trained on regression. The image below is an example of such model (the t subscript in the output layers stands for task). There are several other options to accomplish the same results, so check this <a href="https://arxiv.org/pdf/1707.08114.pdf" rel="nofollow noreferrer">survey</a> as well.</p>
  <p><a href="https://i.sstatic.net/gLH6X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gLH6X.png" alt="enter image description here" /></a></p>
  <p>Last but not least, if multitask learning is not suitable for you, you could simply train 2 individual models, one for action selection and one for repetition prediction. If going down on this route I would suggest to perform classification first and then repetition prediction, since the output of the classification model would be relevant for the regression one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-13T09:38:32.430Z
Answers.LastActivityDate: 2022-01-13T09:38:32.430Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated.</p>
  <p>For example, action = left, n = 3 -&gt; go left 3 times. In this game <span class="math-container">$(left,1)*3 \neq (left,3)$</span> as negative reward is handed out at every single time step (this is for research purposes, so it cannot change).</p>
  <p>I would like to test how a DDQN and a DQN algorithm are affected, as I increase the number of actions available (increase <span class="math-container">$n$</span>).</p>
  <p>My question is; Is there a smarter way to implement this, other than increasing the depth of the output layer? I.e <code>len(output_layer) = n</code>?</p>
  <p>I was thinking of whether or not, there was a way for a single neuron to determine n and then have 4 other neurons that determine the best action? Would this even have any positive effects? (such as less training time, faster computation, better generalization, etc.)</p>
  <p>If yes, how would this typically be done?</p>
  
Tags: <neural-networks><reinforcement-learning><dqn><activation-functions><network-design>
Title: How to output an integer/discrete number n with a single output neuron?
UserDisplayName: GeorgeWTrump
ThreadText: # How to output an integer/discrete number n with a single output neuron?
  
  (Asked by: username_0 on 2022-01-12)
  
  Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated.
  
  For example, action = left, n = 3 -> go left 3 times. In this game $(left,1)\*3 \neq (left,3)$ as negative reward is handed out at every single time step (this is for research purposes, so it cannot change).
  
  I would like to test how a DDQN and a DQN algorithm are affected, as I increase the number of actions available (increase $n$).
  
  My question is; Is there a smarter way to implement this, other than increasing the depth of the output layer? I.e `len(output_layer) = n`?
  
  I was thinking of whether or not, there was a way for a single neuron to determine n and then have 4 other neurons that determine the best action? Would this even have any positive effects? (such as less training time, faster computation, better generalization, etc.)
  
  If yes, how would this typically be done?
  
  **Tags:** neural-networks, reinforcement-learning, dqn, activation-functions, network-design
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-13. Score: 1 | Accepted answer
  
  Predicting the correct amount of repetitions for an action sounds like a **regression** task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being:
  
  * Having to choose a priori a finite max amount of actions n
  * Turning the data into really sparse vectors, especially for large n.
  
  So a better choice in my opinion would be to treat this problem as a regression one, i.e. using a single output node trained on predicting continuous values, and round the predictions to convert real values to integers.
  
  Of course the task of choosing an action is a classification one, but it is possible to combine classification and regression into a single model, using [Multi Task Learning](https://ruder.io/multi-task/), the idea being to train a model with one head (input layer) one body (some hidden layers) and 2 tails (output layers), one trained on classification and one trained on regression. The image below is an example of such model (the t subscript in the output layers stands for task). There are several other options to accomplish the same results, so check this [survey](https://arxiv.org/pdf/1707.08114.pdf) as well.
  
  [![enter image description here](https://i.sstatic.net/gLH6X.png)](https://i.sstatic.net/gLH6X.png)
  
  Last but not least, if multitask learning is not suitable for you, you could simply train 2 individual models, one for action selection and one for repetition prediction. If going down on this route I would suggest to perform classification first and then repetition prediction, since the output of the classification model would be relevant for the regression one.
  
  
  ---
  
  