Answers.Body: <p>What you're describing is known as a composite action space (or a factorisable action space). The key assumption is that the action space can be factorised in sub-action spaces, i.e. <span class="math-container">$\mathcal{A} = \mathcal{A}_1 \times ... \times \mathcal{A}_N$</span>.</p>
  <p>As far as I know, this is a relatively new area in terms of research contributions. The two papers that come to mind are <a href="https://fabiopardo.github.io/posters/action_branching_architectures.pdf" rel="nofollow noreferrer">action branching networks</a> and <a href="https://openreview.net/forum?id=U5XOGxAgccS" rel="nofollow noreferrer">Decoupled Q-Network</a>.</p>
  <p>The latter is likely to be the easiest to implement. Rather than flattening your sub-action spaces into an action space that consists of very many primitive actions and having a DQN provide an output value for the many different possibilities, instead you have <span class="math-container">$N$</span> output heads for each sub-action space. Where a DQN would be a function <span class="math-container">$f : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$</span>, the decoupled network would be <span class="math-container">$N$</span> functions <span class="math-container">$f_i : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}_i|}$</span>. If we refer to the these sub-functions as utility functions, i.e. <span class="math-container">$U_i(s, a_i)$</span>, then we can obtain the Q-value as <span class="math-container">$Q(s, a_1, ..., a_N) = \sum_{i=1}^N U_i(s, a_i)$</span>. This way, the size of the spaces we need to learn values for grows linearly with the number of sub-action spaces rather than exponentially.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-27T10:14:12.063Z
Answers.LastActivityDate: 2023-04-18T10:12:57.290Z
Answers.LastEditDate: 2023-04-18T10:12:57.290Z
Answers.UserDisplayName: David
Body: <p>Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.</p>
  <p>So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions that can be taken for each pet (give food, give water, play).</p>
  <p>Now I know that in the standard approach, the DQN network would be such that the output is of shape 5x3 = 15. But for a larger space of pets and possible actions this would become unfeasible. So is there an adaption of deep q learning in which I could use a network that has only 3+5=8 output layers and treats the selection of the pet and the action independently?</p>
  
Tags: <reinforcement-learning><q-learning><dqn>
Title: Q learning: How to create output layer in which actions are combinations of multiple sub-actions
UserDisplayName: T. Kau
ThreadText: # Q learning: How to create output layer in which actions are combinations of multiple sub-actions
  
  (Asked by: username_0 on 2023-03-27)
  
  Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
  
  So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions that can be taken for each pet (give food, give water, play).
  
  Now I know that in the standard approach, the DQN network would be such that the output is of shape 5x3 = 15. But for a larger space of pets and possible actions this would become unfeasible. So is there an adaption of deep q learning in which I could use a network that has only 3+5=8 output layers and treats the selection of the pet and the action independently?
  
  **Tags:** reinforcement-learning, q-learning, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-27. Score: 3 | Accepted answer
  
  What you're describing is known as a composite action space (or a factorisable action space). The key assumption is that the action space can be factorised in sub-action spaces, i.e. $\mathcal{A} = \mathcal{A}\_1 \times ... \times \mathcal{A}\_N$.
  
  As far as I know, this is a relatively new area in terms of research contributions. The two papers that come to mind are [action branching networks](https://fabiopardo.github.io/posters/action_branching_architectures.pdf) and [Decoupled Q-Network](https://openreview.net/forum?id=U5XOGxAgccS).
  
  The latter is likely to be the easiest to implement. Rather than flattening your sub-action spaces into an action space that consists of very many primitive actions and having a DQN provide an output value for the many different possibilities, instead you have $N$ output heads for each sub-action space. Where a DQN would be a function $f : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, the decoupled network would be $N$ functions $f\_i : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}\_i|}$. If we refer to the these sub-functions as utility functions, i.e. $U\_i(s, a\_i)$, then we can obtain the Q-value as $Q(s, a\_1, ..., a\_N) = \sum\_{i=1}^N U\_i(s, a\_i)$. This way, the size of the spaces we need to learn values for grows linearly with the number of sub-action spaces rather than exponentially.
  
  
  ---
  
  