Answers.Body: <p>I'm not 100% sure whether or not they added any data for terminal game states, but it's very reasonable to indeed make the choice not to include data for terminal game states. As you rightly pointed out, we don't have any meaningful targets to update the policy head towards in those cases, and this is not really a problem because we would also never actually make use of the policy output in a terminal game state. For the value head we <em>could</em> provide meaningful targets to update towards, but again we would never actually have to make use of such outputs; if we encounter a terminal game state in a tree search, we just back up the true value of that terminal game state instead of making a call to the network to obtain a value function approximation.</p>
  <p>In theory, I could imagine some cases where training the value head on terminal game states might be slightly beneficial despite not being strictly necessary; it could enable generalisation to similar game states that are not terminal (but close to being terminal), and speed up learning for those. For example, if you have a game where the goal is to complete a line of <span class="math-container">$5$</span> pieces, training the value head on terminal states where you actually have a line of <span class="math-container">$5$</span> pieces and have entirely won the game might generalise and speed up learning for similar game states where you may not yet have <span class="math-container">$5$</span> pieces in a line, but are very close to that goal. That said, intuitively I really don't feel like this would provide a big benefit (if any), and we could probably also come up with cases where it would be harmful.</p>
  <hr />
  <p>In the <span class="math-container">$\pi^{\text{T}}$</span> notation, <span class="math-container">$\pi$</span> is a vector (for any arbitrary time step, the time step is not specified here) containing a discrete probability distribution over actions (visit counts of MCTS, normalised into a probability distribution), and the <span class="math-container">$\text{T}$</span> simply denotes that we take the transpose of that vector. Personally I don't like the notation though, I prefer something like <span class="math-container">$\pi^{\top}$</span> which is more clearly distinct from a letter <span class="math-container">$T$</span> or <span class="math-container">$\text{T}$</span>.</p>
  <p>Anyway, once you understand that to denote the transpose, you'll see that <span class="math-container">$\pi^{\top}\log(\mathbf{p})$</span> is a dot product between two vectors, which then ends up being a single scalar.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T16:21:14.783Z
Answers.LastActivityDate: 2020-12-16T16:21:14.783Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero.</p>
  <p>From the paper:</p>
  <blockquote>
  <p>The data for each time-step <span class="math-container">$t$</span> is stored as (<span class="math-container">$s_t, \pi_t, z_t$</span>) where <span class="math-container">$z_t = \pm r_T$</span> is the game winner from the perspective of the current player at step <span class="math-container">$t$</span>. In parallel (Figure 1b), new network parameters <span class="math-container">$\Theta_i$</span> are trained from data (<span class="math-container">$s,\pi, z$</span>) sampled uniformly among all time-steps of the last iteration(s) of self-play</p>
  </blockquote>
  <p>Regarding the policy at time <span class="math-container">$t$</span> (<span class="math-container">$\pi_t$</span>), I understood this as the probability distribution of taking some action that is proportional to the visit count to each child node, i.e. during MCTS, given some parent node (state) at time <span class="math-container">$t$</span>, if some child node (subsequent state) <span class="math-container">$a$</span> is visited <span class="math-container">$N_a$</span> times and all children nodes are visited <span class="math-container">$\sum_b N_b$</span> times, then the probability of <span class="math-container">$a$</span> (and its corresponding move) being sampled is <span class="math-container">$\frac{N_a}{\sum_b N_b}$</span>, and this parametrizes the distribution <span class="math-container">$\pi_t$</span>. Is this correct? If this is the case, then for some terminal state <span class="math-container">$T$</span>, we can't parametrize a distribution because we have no children nodes (states) to visit. Does that mean we don't add (<span class="math-container">$s_T, \pi_T, z_T$</span>) to the training data?</p>
  <p>Also, a followup question regarding the loss function:</p>
  <blockquote>
  <p><span class="math-container">$l = (z-v)^2 - \pi^T log\textbf{p} + c||\Theta||^2$</span></p>
  </blockquote>
  <p>I'm confused about this <span class="math-container">$\pi^T$</span> notation. My best guess is that this is a vector of actions sampled from all policies in the <span class="math-container">$N$</span> X <span class="math-container">$(s_t, \pi_t, z_t)$</span> minibatch, but I'm not sure. (PS the <span class="math-container">$T$</span> used in <span class="math-container">$\pi^T$</span> is different from the <span class="math-container">$T$</span> used to denote a terminal state if you look at the paper. Sorry for the confusion, I don't know how to write two different looking T's)</p>
  
Tags: <reinforcement-learning><deep-rl><alphazero><chess><notation>
Title: In AlphaZero, do we need to store the data of terminal states?
UserDisplayName: sb3
ThreadText: # In AlphaZero, do we need to store the data of terminal states?
  
  (Asked by: username_0 on 2020-12-16)
  
  I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero.
  
  From the paper:
  
  > The data for each time-step $t$ is stored as ($s\_t, \pi\_t, z\_t$) where $z\_t = \pm r\_T$ is the game winner from the perspective of the current player at step $t$. In parallel (Figure 1b), new network parameters $\Theta\_i$ are trained from data ($s,\pi, z$) sampled uniformly among all time-steps of the last iteration(s) of self-play
  
  Regarding the policy at time $t$ ($\pi\_t$), I understood this as the probability distribution of taking some action that is proportional to the visit count to each child node, i.e. during MCTS, given some parent node (state) at time $t$, if some child node (subsequent state) $a$ is visited $N\_a$ times and all children nodes are visited $\sum\_b N\_b$ times, then the probability of $a$ (and its corresponding move) being sampled is $\frac{N\_a}{\sum\_b N\_b}$, and this parametrizes the distribution $\pi\_t$. Is this correct? If this is the case, then for some terminal state $T$, we can't parametrize a distribution because we have no children nodes (states) to visit. Does that mean we don't add ($s\_T, \pi\_T, z\_T$) to the training data?
  
  Also, a followup question regarding the loss function:
  
  > $l = (z-v)^2 - \pi^T log\textbf{p} + c||\Theta||^2$
  
  I'm confused about this $\pi^T$ notation. My best guess is that this is a vector of actions sampled from all policies in the $N$ X $(s\_t, \pi\_t, z\_t)$ minibatch, but I'm not sure. (PS the $T$ used in $\pi^T$ is different from the $T$ used to denote a terminal state if you look at the paper. Sorry for the confusion, I don't know how to write two different looking T's)
  
  **Tags:** reinforcement-learning, deep-rl, alphazero, chess, notation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-16. Score: 0 | Accepted answer
  
  I'm not 100% sure whether or not they added any data for terminal game states, but it's very reasonable to indeed make the choice not to include data for terminal game states. As you rightly pointed out, we don't have any meaningful targets to update the policy head towards in those cases, and this is not really a problem because we would also never actually make use of the policy output in a terminal game state. For the value head we *could* provide meaningful targets to update towards, but again we would never actually have to make use of such outputs; if we encounter a terminal game state in a tree search, we just back up the true value of that terminal game state instead of making a call to the network to obtain a value function approximation.
  
  In theory, I could imagine some cases where training the value head on terminal game states might be slightly beneficial despite not being strictly necessary; it could enable generalisation to similar game states that are not terminal (but close to being terminal), and speed up learning for those. For example, if you have a game where the goal is to complete a line of $5$ pieces, training the value head on terminal states where you actually have a line of $5$ pieces and have entirely won the game might generalise and speed up learning for similar game states where you may not yet have $5$ pieces in a line, but are very close to that goal. That said, intuitively I really don't feel like this would provide a big benefit (if any), and we could probably also come up with cases where it would be harmful.
  
  ---
  
  In the $\pi^{\text{T}}$ notation, $\pi$ is a vector (for any arbitrary time step, the time step is not specified here) containing a discrete probability distribution over actions (visit counts of MCTS, normalised into a probability distribution), and the $\text{T}$ simply denotes that we take the transpose of that vector. Personally I don't like the notation though, I prefer something like $\pi^{\top}$ which is more clearly distinct from a letter $T$ or $\text{T}$.
  
  Anyway, once you understand that to denote the transpose, you'll see that $\pi^{\top}\log(\mathbf{p})$ is a dot product between two vectors, which then ends up being a single scalar.
  
  
  ---
  
  