Body: <p>I'm trying to implement the AlphaZero algorithm. AlphaZero uses a neural
  network <span class="math-container">$f_{\theta}$</span> with parameters <span class="math-container">$\theta$</span> for board state <span class="math-container">$s$</span>. It
  returns <span class="math-container">$(p,v)$</span>, where <span class="math-container">$p$</span> is a vector of move probabilities and <span class="math-container">$v$</span> is
  the predicted outcome.</p>
  <p>As far as I know, this is the algorithm:</p>
  <ol>
  <li><p>Initialize random parameters <span class="math-container">$\theta$</span>.</p>
  </li>
  <li><p>Generate a lot of self-play games. For every decision in each game,
  the bot should use MCTS, but instead of rollouts, it uses
  <span class="math-container">$f_{\theta}$</span> as a policy value.</p>
  </li>
  <li><p>Train <span class="math-container">$f_{\theta}$</span> with back-propagation on the games, minimizing
  the loss function.</p>
  </li>
  <li><p>Repeat steps 2-3 with the new <span class="math-container">$f_{\theta}$</span> until the bot is
  sufficiently strong.</p>
  </li>
  </ol>
  <p><strong>I want to understand step 3 better.</strong> This is AlphaZero's loss
  function: <span class="math-container">$$(z - v)^{2} - \mathbf{\pi}^{T}\log p + c\|\theta\|^{2}$$</span></p>
  <p>My belief is that <span class="math-container">$p$</span> and <span class="math-container">$v$</span> are based on <strong>one</strong> board state, while a
  game may have many board states (one for each player's move). But one
  back-propagation step should have the <strong>entire game</strong> as an input, not
  just one board state, right? So how are the resulting <span class="math-container">$p$</span> and <span class="math-container">$v$</span> values
  calculated? Do we take just one board state? Or do we average all of
  them, or take random board states from the game and then average them?
  Or something else?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-01T23:31:09.003Z
Comments.Text: Oh, I think it's the sum of all losses across board states, like the sum in MSE
Comments.UserDisplayName: vbxr
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-02T07:01:07.083Z
Comments.Text: The training does not involve processing entire game as input at once. Instead, it processes individual board states sampled from all self-play games. Training on individual states, rather than averaging over a game, aligns better with the loss function and facilitates efficient optimization using SGD.
Comments.UserDisplayName: cinch
Tags: <deep-learning><game-ai><loss><alphazero><self-play>
Title: How is $(p,v)$ used to calculate loss in the AlphaZero algorithm?
UserDisplayName: vbxr
ThreadText: # How is $(p,v)$ used to calculate loss in the AlphaZero algorithm?
  
  (Asked by: username_0 on 2024-12-01)
  
  I'm trying to implement the AlphaZero algorithm. AlphaZero uses a neural
  network $f\_{\theta}$ with parameters $\theta$ for board state $s$. It
  returns $(p,v)$, where $p$ is a vector of move probabilities and $v$ is
  the predicted outcome.
  
  As far as I know, this is the algorithm:
  
  1. Initialize random parameters $\theta$.
  2. Generate a lot of self-play games. For every decision in each game,
     the bot should use MCTS, but instead of rollouts, it uses
     $f\_{\theta}$ as a policy value.
  3. Train $f\_{\theta}$ with back-propagation on the games, minimizing
     the loss function.
  4. Repeat steps 2-3 with the new $f\_{\theta}$ until the bot is
     sufficiently strong.
  
  **I want to understand step 3 better.** This is AlphaZero's loss
  function: $$(z - v)^{2} - \mathbf{\pi}^{T}\log p + c\|\theta\|^{2}$$
  
  My belief is that $p$ and $v$ are based on **one** board state, while a
  game may have many board states (one for each player's move). But one
  back-propagation step should have the **entire game** as an input, not
  just one board state, right? So how are the resulting $p$ and $v$ values
  calculated? Do we take just one board state? Or do we average all of
  them, or take random board states from the game and then average them?
  Or something else?
  
  **Tags:** deep-learning, game-ai, loss, alphazero, self-play
  
  > **Comments:**
  > **username_0**: Oh, I think it's the sum of all losses across board states, like the sum in MSE  
  > **username_1**: The training does not involve processing entire game as input at once. Instead, it processes individual board states sampled from all self-play games. Training on individual states, rather than averaging over a game, aligns better with the loss function and facilitates efficient optimization using SGD.  
  
  ---
  
  ## 0 answers
  
  