Answers.Body: <ol>
  <li><code>logp</code> seen in code is actually <code>logit p</code> which has this story behind:</li>
  </ol>
  <blockquote>
  <p>Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.</p>
  <p>The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).</p>
  </blockquote>
  <p>Sigmoid near <code>logp</code> is like follows:</p>
  <blockquote>
  <p>The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.</p>
  </blockquote>
  <p>Source: [1]</p>
  <ol start="2">
  <li>In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):</li>
  </ol>
  <blockquote>
  <p>For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).</p>
  </blockquote>
  <ol start="3">
  <li>In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.</li>
  </ol>
  <p>Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.</p>
  <p><strong>Sources:</strong></p>
  <p>[1] <a href="https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/" rel="nofollow noreferrer">https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T15:49:22.107Z
Answers.Text: Thanks a lot for this superb explanation. I do have one question. One other implementation uses one hot encoding to train the neural network but Andrej doesn't do that. Would you be able to be tell me why the difference?
Answers.UserDisplayName: Sarvagya Gupta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T17:50:39.570Z
Answers.Text: Does that another neural network do the same thing (try to play pong)? I don't see how one hot decoding, or any decoding plays part on this task. For categorizing apples and oranges from peaches I see it useful.
Answers.UserDisplayName: mico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T17:59:40.097Z
Answers.Text: See: https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f
Answers.UserDisplayName: mico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T04:35:43.810Z
Answers.Text: So this is Keras implementation of Karpathy's code. This person is also implementing RL on pong. Checkout line 108. I think the person is doing one-hot encoding here
  
  https://github.com/Alexander-H-Liu/Policy-Gradient-and-Actor-Critic-Keras/blob/master/agent_dir/agent_pg.py
Answers.UserDisplayName: Sarvagya Gupta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-22T12:01:14.493Z
Answers.Text: Maybe ask another question about the comparison of these solutions?
Answers.UserDisplayName: mico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T05:51:47.407Z
Answers.Text: can you tell me that I'm the discounted reward function, he adds a reset condition for reward sum and says that it's pong specific. Can you tell me why did he do that?
Answers.UserDisplayName: Sarvagya Gupta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-16T20:36:56.867Z
Answers.LastActivityDate: 2019-11-16T20:36:56.867Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: mico
Body: <p>I was going through the <a href="https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5" rel="nofollow noreferrer">code</a> by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code.</p>
  
  <ol>
  <li><p>Where is the logarithm of the probability being calculated? Nowhere in the code I see him calculating that. </p></li>
  <li><p>Please explain to me the use of <code>dlogps.append(y - aprob)</code> line. I know this is calculating the <code>loss</code>, but how is this helping in a reinforcement learning environment, where we don't have the correct labels?</p></li>
  <li><p>How is <code>policy_backward()</code> working? How are the weights changing to the loss function mentioned above? More specifically, what's <code>dh</code> here?</p></li>
  </ol>
  
Tags: <deep-learning><reinforcement-learning><backpropagation><policy-gradients>
Title: How is gradient being calculated in Andrej Karpathy's pong code?
UserDisplayName: Sarvagya Gupta
ThreadText: # How is gradient being calculated in Andrej Karpathy's pong code?
  
  (Asked by: username_0 on 2019-11-07)
  
  I was going through the [code](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code.
  
  1. Where is the logarithm of the probability being calculated? Nowhere in the code I see him calculating that.
  2. Please explain to me the use of `dlogps.append(y - aprob)` line. I know this is calculating the `loss`, but how is this helping in a reinforcement learning environment, where we don't have the correct labels?
  3. How is `policy_backward()` working? How are the weights changing to the loss function mentioned above? More specifically, what's `dh` here?
  
  **Tags:** deep-learning, reinforcement-learning, backpropagation, policy-gradients
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-16. Score: 1 | Accepted answer
  
  1. `logp` seen in code is actually `logit p` which has this story behind:
  
  > Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.
  >
  > The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).
  
  Sigmoid near `logp` is like follows:
  
  > The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.
  
  Source: [1]
  
  2. In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):
  
  > For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).
  
  3. In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.
  
  Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.
  
  **Sources:**
  
  [1] <https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/>
  
  > **Comments:**
  > **username_0**: Thanks a lot for this superb explanation. I do have one question. One other implementation uses one hot encoding to train the neural network but Andrej doesn't do that. Would you be able to be tell me why the difference?  
  > **username_1**: Does that another neural network do the same thing (try to play pong)? I don't see how one hot decoding, or any decoding plays part on this task. For categorizing apples and oranges from peaches I see it useful.  
  > **username_1**: See: https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f  
  > **username_0**: So this is Keras implementation of Karpathy's code. This person is also implementing RL on pong. Checkout line 108. I think the person is doing one-hot encoding here  https://github.com/Alexander-H-Liu/Policy-Gradient-and-Actor-Critic-Keras/blob/master/agent_dir/agent_pg.py  
  > **username_1**: Maybe ask another question about the comparison of these solutions?  
  > **username_0**: can you tell me that I'm the discounted reward function, he adds a reset condition for reward sum and says that it's pong specific. Can you tell me why did he do that?  
  
  ---
  
  