Answers.Body: <ol>
  <li><p>The bias plane is a layer everywhere equal to the constant <span class="math-container">$a/18$</span> where <span class="math-container">$a$</span> is the action. So each of the 32 frames has three frames for RGB and a fourth frame which is the bias plane making for 128 input layers. This is explained in the Network Architecture section where it mentions that the actions are &quot;broadcast&quot; to the planes.</p>
  </li>
  <li><p>For this I do not have conclusive evidence, but I think tiling a vector means arranging parallel copies of it into e.g. a grid shape. In other words, the input is 6x6x18, and action 1 is represented as all ones in the first plane with all zeros in the remaining planes. One problem with the way you've depicted is that the input is subject to convolutions, but there is no inherent reason why the fifth position and eleventh actions (which are next to each other vertically) should be included in the same 3x3 filter application, but the fifth and seventh (for example) actions should not.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T01:14:45.810Z
Answers.Text: Hi @expz. Thank you for the explanation. I think you are right, but I'm a little confused about why we use different representations for actions in representation and dynamics functions?
Answers.UserDisplayName: Maybe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T03:12:11.333Z
Answers.Text: If you were to use the one hot encoding for the representation network input then there would be 18 planes per frame or 576 planes devoted to actions. I assume they wanted to reduce that to something more manageable. However the fact that they use the one hot encoding instead of the more succinct encoding for the dynamics network which has a similar architecture suggests to me that perhaps the one hot encoding performs better when it is practical to use.
Answers.UserDisplayName: expz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T16:28:32.963Z
Answers.LastActivityDate: 2020-06-20T16:28:32.963Z
Answers.UserDisplayName: expz
Body: <p><a href="https://arxiv.org/abs/1911.08265" rel="nofollow noreferrer">MuZero</a> seems to use two different methods to encode actions into planes for Atari games:</p>
  
  <ol>
  <li>For the input action to the representation function, MuZero encodes historical actions as simple bias planes, scaled as <span class="math-container">$a/18$</span>, where <span class="math-container">$18$</span> is the total number of valid actions in Atari.(from the appendix E of the paper)</li>
  <li>For the input action to the dynamics function, Muzero encode an action as a one-hot vector, which is tiled appropriately into planes(from the appendix F of the paper)</li>
  </ol>
  
  <p>I'm not so sure about how to make of the term "bias plane".</p>
  
  <p>About the second, my understanding is that, as an example, for action <span class="math-container">$4$</span>, we first apply one-hot encoding, which gives us a zero vector of length <span class="math-container">$18$</span> with one in the <span class="math-container">$5$</span>-th position(as there are <span class="math-container">$18$</span> actions). Then we tile it and get a zero vector of length <span class="math-container">$36$</span>, with ones in the <span class="math-container">$5$</span>-th and <span class="math-container">$23$</span>-rd positions. At last, this vector is reshaped into a <span class="math-container">$6\times 6$</span> plane as follows:</p>
  
  <p><span class="math-container">$$
  0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0\\
  0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0
  $$</span></p>
  
Tags: <deep-learning><reinforcement-learning>
Title: How's the action represented in MuZero for Atari?
UserDisplayName: Maybe
ThreadText: # How's the action represented in MuZero for Atari?
  
  (Asked by: username_0 on 2020-05-12)
  
  [MuZero](https://arxiv.org/abs/1911.08265) seems to use two different methods to encode actions into planes for Atari games:
  
  1. For the input action to the representation function, MuZero encodes historical actions as simple bias planes, scaled as $a/18$, where $18$ is the total number of valid actions in Atari.(from the appendix E of the paper)
  2. For the input action to the dynamics function, Muzero encode an action as a one-hot vector, which is tiled appropriately into planes(from the appendix F of the paper)
  
  I'm not so sure about how to make of the term "bias plane".
  
  About the second, my understanding is that, as an example, for action $4$, we first apply one-hot encoding, which gives us a zero vector of length $18$ with one in the $5$-th position(as there are $18$ actions). Then we tile it and get a zero vector of length $36$, with ones in the $5$-th and $23$-rd positions. At last, this vector is reshaped into a $6\times 6$ plane as follows:
  
  $$
  0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0\\
  0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0
  $$
  
  **Tags:** deep-learning, reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-20. Score: 2 | Accepted answer
  
  1. The bias plane is a layer everywhere equal to the constant $a/18$ where $a$ is the action. So each of the 32 frames has three frames for RGB and a fourth frame which is the bias plane making for 128 input layers. This is explained in the Network Architecture section where it mentions that the actions are "broadcast" to the planes.
  2. For this I do not have conclusive evidence, but I think tiling a vector means arranging parallel copies of it into e.g. a grid shape. In other words, the input is 6x6x18, and action 1 is represented as all ones in the first plane with all zeros in the remaining planes. One problem with the way you've depicted is that the input is subject to convolutions, but there is no inherent reason why the fifth position and eleventh actions (which are next to each other vertically) should be included in the same 3x3 filter application, but the fifth and seventh (for example) actions should not.
  
  > **Comments:**
  > **username_0**: Hi @expz. Thank you for the explanation. I think you are right, but I'm a little confused about why we use different representations for actions in representation and dynamics functions?  
  > **username_1**: If you were to use the one hot encoding for the representation network input then there would be 18 planes per frame or 576 planes devoted to actions. I assume they wanted to reduce that to something more manageable. However the fact that they use the one hot encoding instead of the more succinct encoding for the dynamics network which has a similar architecture suggests to me that perhaps the one hot encoding performs better when it is practical to use.  
  
  ---
  
  