Answers.Body: <p>I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the different &quot;factors&quot; they mention in the paper).</p>
  <p>As an example, imagine a game in which you can both move and jump at the same time. In that case, you might have a 4-dimensional discrete action space for moving (NSWE), and a 2-dimensional discrete action space for jumping (yes/no jump), both of which will require a Categorical distribution which has the size of that factor (<span class="math-container">$N_k$</span> in the paper).</p>
  <p>So in this case, you would need to have a categorical distribution for each factor, unless you were to turn these two factors into 1 joint 4*2-dimensional factor and learn a single categorical distribution on that (which would likely be less efficient).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-02T22:24:28.377Z
Answers.LastActivityDate: 2020-12-02T22:24:28.377Z
Answers.UserDisplayName: mdc
Body: <p>I was going through <a href="https://arxiv.org/pdf/1502.05477.pdf" rel="nofollow noreferrer">the TRPO paper</a>, and there was a line under Appendix D &quot;Approximating Factored Policies with Neural Networks&quot; in the last paragraph which I am unable to understand</p>
  <blockquote>
  <p>The action consists of a tuple <span class="math-container">$(a_1, a_2..... , a_K)$</span> of integers <span class="math-container">$a_k\in\{1, 2,......,N_k\} $</span> and each of these components is assumed to have a categorical distribution.</p>
  </blockquote>
  <p>I can't seem to get how each component has a categorical distribution. I think it should be the tuple that has a categorical distribution.
  I think I am getting something wrong.</p>
  
Tags: <reinforcement-learning><papers><trust-region-policy-optimization><action-spaces>
Title: Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?
UserDisplayName: srij
ThreadText: # Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?
  
  (Asked by: username_0 on 2020-11-30)
  
  I was going through [the TRPO paper](https://arxiv.org/pdf/1502.05477.pdf), and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand
  
  > The action consists of a tuple $(a\_1, a\_2..... , a\_K)$ of integers $a\_k\in\{1, 2,......,N\_k\} $ and each of these components is assumed to have a categorical distribution.
  
  I can't seem to get how each component has a categorical distribution. I think it should be the tuple that has a categorical distribution.
  I think I am getting something wrong.
  
  **Tags:** reinforcement-learning, papers, trust-region-policy-optimization, action-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-02. Score: 1 | Accepted answer
  
  I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the different "factors" they mention in the paper).
  
  As an example, imagine a game in which you can both move and jump at the same time. In that case, you might have a 4-dimensional discrete action space for moving (NSWE), and a 2-dimensional discrete action space for jumping (yes/no jump), both of which will require a Categorical distribution which has the size of that factor ($N\_k$ in the paper).
  
  So in this case, you would need to have a categorical distribution for each factor, unless you were to turn these two factors into 1 joint 4\*2-dimensional factor and learn a single categorical distribution on that (which would likely be less efficient).
  
  
  ---
  
  