Answers.Body: <p>An implementation of the 'gradient clipping' can be found in DeepMind's own OpenSpiel.
  <a href="https://github.com/deepmind/open_spiel/blob/master/open_spiel/python/algorithms/neurd.py" rel="nofollow noreferrer">https://github.com/deepmind/open_spiel/blob/master/open_spiel/python/algorithms/neurd.py</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-08T19:07:14.700Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-08T17:46:35.170Z
Answers.LastActivityDate: 2022-09-08T17:46:35.170Z
Answers.UserDisplayName: Ryan Keathley
Body: <p>The paper is here <a href="https://arxiv.org/pdf/1906.00190.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1906.00190.pdf</a></p>
  <p>and the relevant paragraph where they explain their method is below:</p>
  <p><a href="https://i.sstatic.net/Zj8wy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Zj8wy.png" alt="enter image description here" /></a></p>
  <p>It's still not clear to me how this is meant to work exactly. In the pseudo-code, the formula for the new NeuRD policy update is given in the all actions case <span class="math-container">$\theta_{t+1} \leftarrow \theta_{t} + \eta \sum_{a} \nabla_{\theta} y(a) \ A_{\pi}(a)  $</span>, the components of which are just the gradient of each logit (<span class="math-container">$y(a)$</span>) times the advantage of that action. The NeuRD update rule as I understand it is just deep exp3.</p>
  <p>But then is the paragraph from the paper meaning that: for each minibatch and each <span class="math-container">$a, y(a)$</span>, we compute the new model parameters <span class="math-container">$\theta_{t+1, a}$</span> which are given by using only the term in the original update corresponding to <span class="math-container">$a$</span>. Then we see if the logits <span class="math-container">$y_{\theta_{t+1, a}}(a)$</span> of the minibatch forward pass are out of bounds <span class="math-container">$[-B, B]$</span> (either the mean or max logit). If the logit is out of bounds, we do not use that component of the update. Note this not prevent logits from being pushed out of bounds by and update from another action.</p>
  <p>This is my simplest iterpretation of their description and it seems terribly expensive. I'd have to update and un-update the network <span class="math-container">$k$</span> times for each minibatch.</p>
  <p>At least in the tabular case (the NeuRD update rule is just exp3), clipping is necessary to prevent a logit from diverging to <span class="math-container">$- \infty$</span> for an action that is dominated. This context also suggests we can't &quot;cheat&quot; by just considering if the whole update pushes <em>any</em> logit out of bounds, and scrapping the update if so. In the tabular case, this method would eventually zero all gradients once any logit gets pushed out.</p>
  <p>Also I'd like to comment that &quot;gradient clipping&quot; is a poor choice of name. &quot;logit clipping&quot; seems from accurate.</p>
  
Tags: <policy-gradients><pytorch><research><deepmind>
Title: How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper?
UserDisplayName: Ryan Keathley
ThreadText: # How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper?
  
  (Asked by: username_0 on 2022-09-07)
  
  The paper is here <https://arxiv.org/pdf/1906.00190.pdf>
  
  and the relevant paragraph where they explain their method is below:
  
  [![enter image description here](https://i.sstatic.net/Zj8wy.png)](https://i.sstatic.net/Zj8wy.png)
  
  It's still not clear to me how this is meant to work exactly. In the pseudo-code, the formula for the new NeuRD policy update is given in the all actions case $\theta\_{t+1} \leftarrow \theta\_{t} + \eta \sum\_{a} \nabla\_{\theta} y(a) \ A\_{\pi}(a) $, the components of which are just the gradient of each logit ($y(a)$) times the advantage of that action. The NeuRD update rule as I understand it is just deep exp3.
  
  But then is the paragraph from the paper meaning that: for each minibatch and each $a, y(a)$, we compute the new model parameters $\theta\_{t+1, a}$ which are given by using only the term in the original update corresponding to $a$. Then we see if the logits $y\_{\theta\_{t+1, a}}(a)$ of the minibatch forward pass are out of bounds $[-B, B]$ (either the mean or max logit). If the logit is out of bounds, we do not use that component of the update. Note this not prevent logits from being pushed out of bounds by and update from another action.
  
  This is my simplest iterpretation of their description and it seems terribly expensive. I'd have to update and un-update the network $k$ times for each minibatch.
  
  At least in the tabular case (the NeuRD update rule is just exp3), clipping is necessary to prevent a logit from diverging to $- \infty$ for an action that is dominated. This context also suggests we can't "cheat" by just considering if the whole update pushes *any* logit out of bounds, and scrapping the update if so. In the tabular case, this method would eventually zero all gradients once any logit gets pushed out.
  
  Also I'd like to comment that "gradient clipping" is a poor choice of name. "logit clipping" seems from accurate.
  
  **Tags:** policy-gradients, pytorch, research, deepmind
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-09-08. Score: 1 | Accepted answer
  
  An implementation of the 'gradient clipping' can be found in DeepMind's own OpenSpiel.
  <https://github.com/deepmind/open_spiel/blob/master/open_spiel/python/algorithms/neurd.py>
  
  > **Comments:**
  > **username_1**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  