Body: <p>In section 3 of <a href="https://web.eecs.umich.edu/~baveja/Papers/IJCAIGSLL2016.pdf" rel="nofollow noreferrer">this paper</a> the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. </p>
  
  <p>I have read it a number of times and understand it up through the end of the explanation of GARB. The author then goes on to explain how they use backprop to implement this procedure, which is the point at which I stop understanding. </p>
  
  <p><a href="https://i.sstatic.net/InOjt.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/InOjt.jpg" alt="pic1"></a>
  <a href="https://i.sstatic.net/wW79K.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wW79K.jpg" alt="pic2"></a></p>
  
  <p>Is there an open source implementation available to look at? I can’t figure out if <span class="math-container">$g_T$</span> is actually computed and used or not? And not certain how the internal reward gradient is calculated. </p>
  
  <p>Any insight you can provide would be helpful. </p>
  
  <p><strong>EDIT:</strong> after reading it several times and several related papers, I think I have more understanding but not quite there. So my main questions are:</p>
  
  <p>1) are we keeping a full eligibility trace vector with the dimensionality of the vector equal to the number of parameters in <span class="math-container">$\theta$</span> (all NN params)?</p>
  
  <p>2) do we use the gradient calculation via backdrop at every step to calculate <span class="math-container">$g_t$</span>?</p>
  
  <p>3) do we have to maintain <span class="math-container">$3 * \theta$</span> parameters, one for theta, one for <span class="math-container">$e$</span> and one for <span class="math-container">$g_t$</span>?</p>
  
  <p>3) what then is the procedure at terminal <span class="math-container">$g_T$</span> to update the parameters? A simple element wise matrix operation?</p>
  
  <p>4) How often to update the parameters of <span class="math-container">$\theta$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-06T14:43:59.367Z
Comments.Text: Hi Hanzy! Please, ask one question per post. You're asking too many questions and thus you will make the life of possible answerers harder. Split this post into multiple ones: one post for each of your doubt, unless the questions are very related.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-06T14:48:28.397Z
Comments.Text: @nbro ok will do. My concern is I don’t know how widely read the paper is, so wasn’t sure if it would be helpful or not to break it up. I’ll edit later today.
Comments.UserDisplayName: Hanzy
Tags: <neural-networks><reinforcement-learning><monte-carlo-tree-search><rewards><monte-carlo-methods>
Title: How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?
UserDisplayName: Hanzy
ThreadText: # How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?
  
  (Asked by: username_0 on 2019-05-05)
  
  In section 3 of [this paper](https://web.eecs.umich.edu/~baveja/Papers/IJCAIGSLL2016.pdf) the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator.
  
  I have read it a number of times and understand it up through the end of the explanation of GARB. The author then goes on to explain how they use backprop to implement this procedure, which is the point at which I stop understanding.
  
  [![pic1](https://i.sstatic.net/InOjt.jpg)](https://i.sstatic.net/InOjt.jpg)
  [![pic2](https://i.sstatic.net/wW79K.jpg)](https://i.sstatic.net/wW79K.jpg)
  
  Is there an open source implementation available to look at? I can’t figure out if $g\_T$ is actually computed and used or not? And not certain how the internal reward gradient is calculated.
  
  Any insight you can provide would be helpful.
  
  **EDIT:** after reading it several times and several related papers, I think I have more understanding but not quite there. So my main questions are:
  
  1) are we keeping a full eligibility trace vector with the dimensionality of the vector equal to the number of parameters in $\theta$ (all NN params)?
  
  2) do we use the gradient calculation via backdrop at every step to calculate $g\_t$?
  
  3) do we have to maintain $3 \* \theta$ parameters, one for theta, one for $e$ and one for $g\_t$?
  
  3) what then is the procedure at terminal $g\_T$ to update the parameters? A simple element wise matrix operation?
  
  4) How often to update the parameters of $\theta$?
  
  **Tags:** neural-networks, reinforcement-learning, monte-carlo-tree-search, rewards, monte-carlo-methods
  > **Comments:**
  > **username_1**: Hi Hanzy! Please, ask one question per post. You're asking too many questions and thus you will make the life of possible answerers harder. Split this post into multiple ones: one post for each of your doubt, unless the questions are very related.  
  > **username_0**: @nbro ok will do. My concern is I don’t know how widely read the paper is, so wasn’t sure if it would be helpful or not to break it up. I’ll edit later today.  
  
  ---
  
  ## 0 answers
  
  