title: Understanding Gradient Policy Deriving
body: <p>I'm trying to recreate very simple example of Policy Gradient, from it's origin resource <a href="http://karpathy.github.io/2016/05/31/rl/" rel="nofollow noreferrer">Andrej Karpathy blog</a>. In that articale, you will find example with CartPole and Policy Gradient with list of weight and Softmax activation. Here is my recreated and very simple example of CartPole policy gradient, <strong>which works perfect</strong>.</p>
  <pre><code>import gym
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn.preprocessing import PolynomialFeatures
  import copy
  
  NUM_EPISODES = 4000
  LEARNING_RATE = 0.000025
  GAMMA = 0.99
  
  
  # noinspection PyMethodMayBeStatic
  class Agent:
      def __init__(self):
          self.poly = PolynomialFeatures(1)
          self.w = np.random.rand(5, 2)
  
      def policy(self, state):
          z = state.dot(self.w)
          exp = np.exp(z)
          return exp/np.sum(exp)
  
      def __softmax_grad(self, softmax):
          s = softmax.reshape(-1,1)
          return np.diagflat(s) - np.dot(s, s.T)
  
      def grad(self, probs, action, state):
          dsoftmax = self.__softmax_grad(probs)[action,:]
          dlog = dsoftmax / probs[0,action]
          grad = state.T.dot(dlog[None,:])
          return grad
  
      def update_with(self, grads, rewards):
  
          for i in range(len(grads)):
              # Loop through everything that happend in the episode
              # and update towards the log policy gradient times **FUTURE** reward
  
              total_grad_effect = 0
              for t, r in enumerate(rewards[i:]):
                  total_grad_effect += r * (GAMMA ** r)
              self.w += LEARNING_RATE * grads[i] * total_grad_effect
              print(&quot;Grads update: &quot; + str(np.sum(grads[i])))
  
  
  
  def main(argv):
      env = gym.make('CartPole-v0')
      np.random.seed(1)
  
      agent = Agent()
      complete_scores = []
  
      for e in range(NUM_EPISODES):
          state = env.reset()[None, :]
          state = agent.poly.fit_transform(state)
  
          rewards = []
          grads = []
          score = 0
  
          while True:
  
              probs = agent.policy(state)
              action_space = env.action_space.n
              action = np.random.choice(action_space, p=probs[0])
  
              next_state, reward, done,_ = env.step(action)
              next_state = next_state[None,:]
              next_state = agent.poly.fit_transform(next_state.reshape(1, 4))
              grad = agent.grad(probs, action, state)
  
              grads.append(grad)
              rewards.append(reward)
  
              score += reward
              state = next_state
  
              if done:
                  break
  
          agent.update_with(grads, rewards)
          complete_scores.append(score)
  
      env.close()
      plt.plot(np.arange(NUM_EPISODES),
               complete_scores)
      plt.savefig('image1.png')
  
  
  if __name__ == '__main__':
      main(None)
  </code></pre>
  <p><a href="https://i.stack.imgur.com/m2oBc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m2oBc.png" alt="enter image description here" /></a></p>
  <p>.</p>
  <p>.</p>
  <p><strong>Question</strong></p>
  <p>I'm trying to do, almost the same example but with Sigmoid activation (just for simplicity). <strong>That is all I need to do. Switch activation in the model from <code>softmax</code> to the <code>sigmoid</code>.</strong> Which should work for sure (based on explanation below). But my Policy Gradient model don't learn anything and keeps random. Any suggestion?</p>
  <pre><code>import gym
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn.preprocessing import PolynomialFeatures
  
  NUM_EPISODES = 4000
  LEARNING_RATE = 0.000025
  GAMMA = 0.99
  
  
  # noinspection PyMethodMayBeStatic
  class Agent:
      def __init__(self):
          self.poly = PolynomialFeatures(1)
          self.w = np.random.rand(5, 1) - 0.5
  
      # Our policy that maps state to action parameterized by w
      # noinspection PyShadowingNames
      def policy(self, state):
          z = np.sum(state.dot(self.w))
          return self.sigmoid(z)
  
      def sigmoid(self, x):
          s = 1 / (1 + np.exp(-x))
          return s
  
      def sigmoid_grad(self, sig_x):
          return sig_x * (1 - sig_x)
  
      def grad(self, probs, action, state):
          dsoftmax = self.sigmoid_grad(probs)
          dlog = dsoftmax / probs
          grad = state.T.dot(dlog)
          grad = grad.reshape(5, 1)
          return grad
  
      def update_with(self, grads, rewards):
          if len(grads) &lt; 50:
              return
          for i in range(len(grads)):
              # Loop through everything that happened in the episode
              # and update towards the log policy gradient times **FUTURE** reward
  
              total_grad_effect = 0
              for t, r in enumerate(rewards[i:]):
                  total_grad_effect += r * (GAMMA ** r)
              self.w += LEARNING_RATE * grads[i] * total_grad_effect
  
  
  def main(argv):
      env = gym.make('CartPole-v0')
      np.random.seed(1)
  
      agent = Agent()
      complete_scores = []
  
      for e in range(NUM_EPISODES):
          state = env.reset()[None, :]
          state = agent.poly.fit_transform(state)
  
          rewards = []
          grads = []
          score = 0
  
          while True:
  
              probs = agent.policy(state)
              action_space = env.action_space.n
              action = np.random.choice(action_space, p=[1 - probs, probs])
  
              next_state, reward, done, _ = env.step(action)
              next_state = next_state[None, :]
              next_state = agent.poly.fit_transform(next_state.reshape(1, 4))
  
              grad = agent.grad(probs, action, state)
              grads.append(grad)
              rewards.append(reward)
  
              score += reward
              state = next_state
  
              if done:
                  break
  
          agent.update_with(grads, rewards)
          complete_scores.append(score)
  
      env.close()
      plt.plot(np.arange(NUM_EPISODES),
               complete_scores)
      plt.savefig('image1.png')
  
  
  if __name__ == '__main__':
      main(None)
  </code></pre>
  <p>Plotting all the learning keeps random. Nothing helps with tuning hyper parameters. Below the sample image.</p>
  <p><a href="https://i.stack.imgur.com/LEiVk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LEiVk.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p><strong>References</strong>:</p>
  </blockquote>
  <blockquote>
  <ol>
  <li><a href="http://karpathy.github.io/2016/05/31/rl/" rel="nofollow noreferrer"><strong>Deep Reinforcement Learning: Pong from Pixels</strong></a></li>
  </ol>
  </blockquote>
  <blockquote>
  <ol start="2">
  <li><a href="https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/" rel="nofollow noreferrer"><strong>An introduction to Policy Gradients with Cartpole and Doom</strong></a></li>
  </ol>
  </blockquote>
  <blockquote>
  <ol start="3">
  <li><a href="https://medium.com/@thechrisyoon/deriving-policy-gradients-and-implementing-reinforce-f887949bd63" rel="nofollow noreferrer"><strong>Deriving Policy Gradients and Implementing REINFORCE</strong></a></li>
  </ol>
  </blockquote>
  <blockquote>
  <ol start="4">
  <li><a href="http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/" rel="nofollow noreferrer"><strong>Machine Learning Trick of the Day (5): Log Derivative Trick 12</strong></a></li>
  </ol>
  </blockquote>
  <hr />
  <p><strong>UPDATE</strong></p>
  <p>Seems like answer below could doing some work from the graphic. But it's Not log probability, and Not even gradient of the policy. And changes whole purpose of the RL Gradient Policy. Please check references above. Following the image we next statement.</p>
  <p><a href="https://i.stack.imgur.com/frprX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/frprX.png" alt="enter image description here" /></a></p>
  <p>I need to take a <strong>Gradient of the Log function of my Policy</strong> (which is simply weights and <code>sigmoid</code> activation). Please let me know in case of any questions.</p>