Body: <p>I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (<a href="https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5" rel="nofollow noreferrer">https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5</a>) to 3 possible actions (up/down/stay instead of up/down).</p>
  <p>I changed the policy_forward() so that it ends with a softmax and not sigmoid since there are 3 probs for the 3 actions now. However, I have some difficulties calculating the gradient. I don't know whether it should be y-aprob (used in Andrej's version with 2 actions where one neuron is sufficient - high activation=UP, low activation=DOWN) or aprob-y (or maybe even something else). y is the one hot encoded vector of the taken action (e.g. [0, 0, 1]) and aprob are the post-softmax probabilities of the actions (e.g. [0.1, 0.25, 0.65]).</p>
  <p>After doing some research, it seems to me that aprob-y is the correct derivative of the cross entropy loss. The problem is, the network doesn't learn at all - actually I think it starts to perform worse and worse (just like if it was trying to minimize the reward?). At the beginning, the agent sometimes manages to score a point, but after some learning, the agent just loses every single time. The network with y-aprob seems to work though and I have no idea why, because the formula is different than what I found on the internet (actually I found one piece of code on github that also uses y-aprob: <a href="https://github.com/berkott/SpaceInvadersAI/blob/master/Pong/pong_simple.py" rel="nofollow noreferrer">https://github.com/berkott/SpaceInvadersAI/blob/master/Pong/pong_simple.py</a> ---&gt; &quot;gradients = stacked_actions - stacked_predictions&quot;).</p>
  <p>Does anyone have an idea of what the gradient should be and why? I would especially love to hear the &quot;why&quot; part because for now I am using the y-aprob gradient but have completely no intuition on why it's more or less working.</p>
  <p>Forward pass:</p>
  <pre><code>def policy_forward(x):
    # shapes -&gt; numpy matmul docs
    h = x @ model['W1'] # (6400,) @ (6400, 300) ---&gt; (1, 6400) @ (6400, 300) ---&gt; (1, 300) ---&gt; (300,) 
    h[h&lt;0] = 0 # ReLU nonlinearity
    logits = h @ model['W2'] # (300,) @ (300, 3) ---&gt; (1, 300) @ (300, 3) ---&gt; (1, 3) ---&gt; (3,)
    p = softmax(logits) # instead of the sigmoid
    return p, h # return probability of taking each of the 3 actions and the hidden states
  </code></pre>
  <p>Confusing backprop part:</p>
  <pre><code>RIGHT, LEFT = 2, 3
  
  # forward the policy network and sample an action from the returned probability
  aprob, h = policy_forward(x)
  action = random.choices(range(len(aprob)), weights=aprob, k=1)
  
  # record various intermediates (needed later for backprop)
  xs.append(x) # observation
  hs.append(h) # hidden state
  
  # cross entropy loss derivative
  y = np.zeros_like(aprob)
  y[action] = 1
  dlogps.append(y - aprob)
  
  if action[0] == 0: a = RIGHT
  elif action[0] == 1: a = LEFT
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T12:13:13.890Z
Comments.Text: in policy gradient you need to maximise the log-likelihood of your chosen action, weighted by the returns/advantage. minimising the cross entropy is equivalent to maximising the log-likelihood, so calculate your cross entropy, weight by the returns/advantage, and then minimise.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T12:27:36.440Z
Comments.Text: Oh, so the difference in the sign might be because my parameter update looks like that: `model[k] += learning_rate * g / (np.sqrt(rmsprop_cache[k]) + 1e-5)`? I'm moving in the positive gradient direction (because of the +=), so the loss, in order for it to be minimized, needs to have a negative sign? And that's how it becomes y-aprob instead of aprob-y, which is the actual cross entropy loss derivative? Btw. I do multiply the gradient with the advantage, just forgot to add the code snippet here
Comments.UserDisplayName: Blato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T14:54:30.053Z
Comments.Text: honestly, I did not read your code, I was more telling you what your code should do. looking at your code, I can't see where you are taking the log of aprob? looks like you take the softmax and never log it.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T17:08:07.833Z
Comments.Text: Ohh, yeah, for some reason I thought it wouldn't be necessary. But now that I'm reading about it, I should probably take the log of aprob. Where should I do it though? Doing `dlogps.append(np.log(aprob) - y)` causes the model not to learn anything and causes an error after some episodes: `action = random.choices(range(len(aprob)), weights=aprob, k=1)
    File "/usr/lib/python3.10/random.py", line 537, in choices
      raise ValueError('Total of weights must be finite')
  ValueError: Total of weights must be finite`. Maybe some probability was 0 and the log jumped to infinity
Comments.UserDisplayName: Blato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T17:13:58.360Z
Comments.Text: Or actually, do I need the log? `aprob - y` is already the derivative of the cross entropy loss (or at least I think so). I only calculate the softmax, because I need softmax probabilities to choose an action. I could later calculate the actual loss (I would have to take the negative log of the softmax probs, right? and later maybe the mean value?) but I don't think need it. I only need its derivative for the backprop
Comments.UserDisplayName: Blato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T17:53:59.173Z
Comments.Text: if you are calculating the derivative 'by hand' then you may not need to compute the log. though, if you are compute the derivative by hand, why not just differentiate log of the policy and then perform a gradient ascent step using this gradient (scaled by the advantage)?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-06T18:41:25.380Z
Comments.Text: I'm not sure if I know what you mean by that. `aprob - y` is the derivative of the cross entropy loss which already contains the log of the softmax output. I'm just not calculating the loss itself because I don't need it, the derivative is enough for the backprop. I think that I'm using the same formula as in this lecture: https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec21.pdf (slide 13, last line of "To apply REINFORCE..."). I might be wrong though
Comments.UserDisplayName: Blato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-07T14:19:48.443Z
Comments.Text: cross entropy loss is defined as $-\log p_\theta$, where $p_\theta$ is the predicted probability of the _true_ class distribution. differentiating this wrt $\theta$ gives $-\frac{\nabla_\theta p_\theta}{p_\theta}$. that is your gradient of the cross entropy loss. how does this relate to aprob? is aprob the derivative of your weights? to me it looks like it is just the probability value of the selected action. can you clarify?
Comments.UserDisplayName: David
Tags: <reinforcement-learning><backpropagation><policy-gradients><gym><cross-entropy>
Title: Reinforcement learning - calculating policy gradient using cross entropy loss
UserDisplayName: Blato
ThreadText: # Reinforcement learning - calculating policy gradient using cross entropy loss
  
  (Asked by: username_0 on 2024-03-06)
  
  I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) to 3 possible actions (up/down/stay instead of up/down).
  I changed the policy_forward() so that it ends with a softmax and not sigmoid since there are 3 probs for the 3 actions now. However, I have some difficulties calculating the gradient. I don't know whether it should be y-aprob (used in Andrej's version with 2 actions where one neuron is sufficient - high activation=UP, low activation=DOWN) or aprob-y (or maybe even something else). y is the one hot encoded vector of the taken action (e.g. [0, 0, 1]) and aprob are the post-softmax probabilities of the actions (e.g. [0.1, 0.25, 0.65]).
  After doing some research, it seems to me that aprob-y is the correct derivative of the cross entropy loss. The problem is, the network doesn't learn at all - actually I think it starts to perform worse and worse (just like if it was trying to minimize the reward?). At the beginning, the agent sometimes manages to score a point, but after some learning, the agent just loses every single time. The network with y-aprob seems to work though and I have no idea why, because the formula is different than what I found on the internet (actually I found one piece of code on github that also uses y-aprob: https://github.com/berkott/SpaceInvadersAI/blob/master/Pong/pong_simple.py ---> "gradients = stacked_actions - stacked_predictions").
  Does anyone have an idea of what the gradient should be and why? I would especially love to hear the "why" part because for now I am using the y-aprob gradient but have completely no intuition on why it's more or less working.
  Forward pass:
  def policy_forward(x):
    # shapes -> numpy matmul docs
    h = x @ model['W1'] # (6400,) @ (6400, 300) ---> (1, 6400) @ (6400, 300) ---> (1, 300) ---> (300,) 
    h[h<0] = 0 # ReLU nonlinearity
    logits = h @ model['W2'] # (300,) @ (300, 3) ---> (1, 300) @ (300, 3) ---> (1, 3) ---> (3,)
    p = softmax(logits) # instead of the sigmoid
    return p, h # return probability of taking each of the 3 actions and the hidden states
  
  Confusing backprop part:
  RIGHT, LEFT = 2, 3
  
  # forward the policy network and sample an action from the returned probability
  aprob, h = policy_forward(x)
  action = random.choices(range(len(aprob)), weights=aprob, k=1)
  
  # record various intermediates (needed later for backprop)
  xs.append(x) # observation
  hs.append(h) # hidden state
  
  # cross entropy loss derivative
  y = np.zeros_like(aprob)
  y[action] = 1
  dlogps.append(y - aprob)
  
  if action[0] == 0: a = RIGHT
  elif action[0] == 1: a = LEFT
  
  **Tags:** reinforcement-learning, backpropagation, policy-gradients, gym, cross-entropy
  > **Comments:**
  > **username_1**: in policy gradient you need to maximise the log-likelihood of your chosen action, weighted by the returns/advantage. minimising the cross entropy is equivalent to maximising the log-likelihood, so calculate your cross entropy, weight by the returns/advantage, and then minimise.  
  > **username_0**: Oh, so the difference in the sign might be because my parameter update looks like that: `model[k] += learning_rate * g / (np.sqrt(rmsprop_cache[k]) + 1e-5)`? I'm moving in the positive gradient direction (because of the +=), so the loss, in order for it to be minimized, needs to have a negative sign? And that's how it becomes y-aprob instead of aprob-y, which is the actual cross entropy loss derivative? Btw. I do multiply the gradient with the advantage, just forgot to add the code snippet here  
  > **username_1**: honestly, I did not read your code, I was more telling you what your code should do. looking at your code, I can't see where you are taking the log of aprob? looks like you take the softmax and never log it.  
  > **username_0**: Ohh, yeah, for some reason I thought it wouldn't be necessary. But now that I'm reading about it, I should probably take the log of aprob. Where should I do it though? Doing `dlogps.append(np.log(aprob) - y)` causes the model not to learn anything and causes an error after some episodes: `action = random.choices(range(len(aprob)), weights=aprob, k=1)   File "/usr/lib/python3.10/random.py", line 537, in choices     raise ValueError('Total of weights must be finite') ValueError: Total of weights must be finite`. Maybe some probability was 0 and the log jumped to infinity  
  > **username_0**: Or actually, do I need the log? `aprob - y` is already the derivative of the cross entropy loss (or at least I think so). I only calculate the softmax, because I need softmax probabilities to choose an action. I could later calculate the actual loss (I would have to take the negative log of the softmax probs, right? and later maybe the mean value?) but I don't think need it. I only need its derivative for the backprop  
  > **username_1**: if you are calculating the derivative 'by hand' then you may not need to compute the log. though, if you are compute the derivative by hand, why not just differentiate log of the policy and then perform a gradient ascent step using this gradient (scaled by the advantage)?  
  > **username_0**: I'm not sure if I know what you mean by that. `aprob - y` is the derivative of the cross entropy loss which already contains the log of the softmax output. I'm just not calculating the loss itself because I don't need it, the derivative is enough for the backprop. I think that I'm using the same formula as in this lecture: https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec21.pdf (slide 13, last line of "To apply REINFORCE..."). I might be wrong though  
  > **username_1**: cross entropy loss is defined as $-\log p_\theta$, where $p_\theta$ is the predicted probability of the _true_ class distribution. differentiating this wrt $\theta$ gives $-\frac{\nabla_\theta p_\theta}{p_\theta}$. that is your gradient of the cross entropy loss. how does this relate to aprob? is aprob the derivative of your weights? to me it looks like it is just the probability value of the selected action. can you clarify?  
  
  ---
  
  ## 0 answers
  
  