Answers.Body: <p>They are not maximizing the gradient, the gradient is of the form
  <span class="math-container">\begin{equation}
  \nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t))
  \end{equation}</span>
  that means that when implementing it in software you can form your objective as
  <span class="math-container">\begin{equation}
  J = \sum_{t=0}^T G_t \log(\pi_{\theta}(a_t|s_t))
  \end{equation}</span>
  and then taking the gradient of that objective is equal to the policy gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T11:02:03.107Z
Answers.Text: I appreciate you taking the time to answer my naive question!
Answers.UserDisplayName: Hao Huynh Nhat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T16:30:50.427Z
Answers.Text: @HaoHuynhNhat you should upvote and accept the answer if it has answered your question.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T02:52:07.610Z
Answers.Text: @DavidIreland Yeah thanks for your reminder! this is my first question so I didn't know that!
Answers.UserDisplayName: Hao Huynh Nhat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T09:33:39.093Z
Answers.LastActivityDate: 2021-04-05T09:33:39.093Z
Answers.UserDisplayName: Brale
Body: <p>I read the book &quot;Foundation of Deep Reinforcement Learning,  Laura Graesser and Wah Loon Keng&quot;, and when I go through the REINFORCE algorithm, they show the objective function:</p>
  <p><span class="math-container">$$
  J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}[R(\tau)]=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} \gamma^{t} r_{t}\right]
  $$</span></p>
  <p>and the gradient of the objective:</p>
  <p><span class="math-container">$$
  \nabla_{\theta} J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} R_{t}(\tau) \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right)\right]
  $$</span></p>
  <p>But when they implement it,</p>
  <pre><code>class Pi(nn.Module):
      def __init__(self, in_dim, out_dim):
          super(Pi, self).__init__()
          layers = [
                  nn.Linear(in_dim, 64),
                  nn.ReLU(),
                  nn.Linear(64, out_dim)
          ]
          self.model = nn.Sequential(*layers)
          self.onpolicy_reset()
          self.train()
  
      def onpolicy_reset(self):
          self.log_probs = []
          self.rewards = []
  
      def forward(self, x):
          pdparam = self.model(x)
          return pdparam
  
      def act(self, state):
          x = torch.from_numpy(state.astype(np.float32))
          pdparam = self.forward(x) # (1, num_action), each number represent the raw logits for that specific action
          # model contain the paremeters theta of the policy, pd is the probability 
          # distribution parameterized by model's theta 
          pd = Categorical(logits = pdparam)
          action = pd.sample()
          log_prob = pd.log_prob(action)
          self.log_probs.append(log_prob)
          return action.item()
  
  def train(pi, optimizer):
      T = len(pi.rewards)
      rets = np.empty(T, dtype = np.float32)
      future_ret = 0.0
      for t in reversed(range(T)):
          future_ret = pi.rewards[t] + gamma*future_ret
          rets[t] = future_ret
  
      rets = torch.tensor(rets)
      log_probs = torch.stack(pi.log_probs)
      loss = -log_probs*rets
      loss = torch.sum(loss)
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      return loss
  
  def main():
      env = gym.make('CartPole-v0')
      # in_dim is the state dimension
      in_dim = env.observation_space.shape[0]
      # out_dim is the action dimension
      out_dim = env.action_space.n
      pi = Pi(in_dim, out_dim)
      optimizer = optim.Adam(pi.parameters(), lr = 0.005)
      for epi in range(300):
          state = env.reset()
          for t in range(200): # max timstep of cartpole is 200
              action = pi.act(state)
              state, reward, done, _ = env.step(action)
              pi.rewards.append(reward)
          # env.render(mode='rgb_array')
              if done:
                  break
          loss = train(pi, optimizer)
          total_reward = sum(pi.rewards)
          solved = total_reward &gt; 195.0
          pi.onpolicy_reset()
          print(f'Episode {epi}, loss: {loss}, total reward: {total_reward}, solve: {solved}')
      return pi
  </code></pre>
  <p>In <code>train()</code>, they minimize the gradient term, and I can not understand why is that.</p>
  <p>Can someone shed light on that?</p>
  <p>I am new to this so please forget me if this question is stupid.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-02T09:28:48.357Z
Comments.Text: Because we want to maximise the objective $J$, i.e we want to maximise expected future returns.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-05T06:55:20.467Z
Comments.Text: I am sorry for some confusion in the question, where I ask "why do they minimize the gradient but not the loss", but actually they maximize the gradient of the objective. However, the point is why the `gradient` but not the `objective` function to be maximized?
Comments.UserDisplayName: Hao Huynh Nhat
Tags: <reinforcement-learning><objective-functions><policy-gradients><reinforce>
Title: Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?
UserDisplayName: Hao Huynh Nhat
ThreadText: # Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?
  
  (Asked by: username_0 on 2021-04-02)
  
  I read the book "Foundation of Deep Reinforcement Learning,  Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function:
  $$
  J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}[R(\tau)]=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} \gamma^{t} r_{t}\right]
  $$
  and the gradient of the objective:
  $$
  \nabla_{\theta} J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} R_{t}(\tau) \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right)\right]
  $$
  But when they implement it,
  class Pi(nn.Module):
      def __init__(self, in_dim, out_dim):
          super(Pi, self).__init__()
          layers = [
                  nn.Linear(in_dim, 64),
                  nn.ReLU(),
                  nn.Linear(64, out_dim)
          ]
          self.model = nn.Sequential(*layers)
          self.onpolicy_reset()
          self.train()
  
      def onpolicy_reset(self):
          self.log_probs = []
          self.rewards = []
  
      def forward(self, x):
          pdparam = self.model(x)
          return pdparam
  
      def act(self, state):
          x = torch.from_numpy(state.astype(np.float32))
          pdparam = self.forward(x) # (1, num_action), each number represent the raw logits for that specific action
          # model contain the paremeters theta of the policy, pd is the probability 
          # distribution parameterized by model's theta 
          pd = Categorical(logits = pdparam)
          action = pd.sample()
          log_prob = pd.log_prob(action)
          self.log_probs.append(log_prob)
          return action.item()
  
  def train(pi, optimizer):
      T = len(pi.rewards)
      rets = np.empty(T, dtype = np.float32)
      future_ret = 0.0
      for t in reversed(range(T)):
          future_ret = pi.rewards[t] + gamma*future_ret
          rets[t] = future_ret
  
      rets = torch.tensor(rets)
      log_probs = torch.stack(pi.log_probs)
      loss = -log_probs*rets
      loss = torch.sum(loss)
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      return loss
  
  def main():
      env = gym.make('CartPole-v0')
      # in_dim is the state dimension
      in_dim = env.observation_space.shape[0]
      # out_dim is the action dimension
      out_dim = env.action_space.n
      pi = Pi(in_dim, out_dim)
      optimizer = optim.Adam(pi.parameters(), lr = 0.005)
      for epi in range(300):
          state = env.reset()
          for t in range(200): # max timstep of cartpole is 200
              action = pi.act(state)
              state, reward, done, _ = env.step(action)
              pi.rewards.append(reward)
          # env.render(mode='rgb_array')
              if done:
                  break
          loss = train(pi, optimizer)
          total_reward = sum(pi.rewards)
          solved = total_reward > 195.0
          pi.onpolicy_reset()
          print(f'Episode {epi}, loss: {loss}, total reward: {total_reward}, solve: {solved}')
      return pi
  
  In train(), they minimize the gradient term, and I can not understand why is that.
  Can someone shed light on that?
  I am new to this so please forget me if this question is stupid.
  
  **Tags:** reinforcement-learning, objective-functions, policy-gradients, reinforce
  > **Comments:**
  > **username_1**: Because we want to maximise the objective $J$, i.e we want to maximise expected future returns.  
  > **username_0**: I am sorry for some confusion in the question, where I ask "why do they minimize the gradient but not the loss", but actually they maximize the gradient of the objective. However, the point is why the `gradient` but not the `objective` function to be maximized?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-04-05. Score: 1 | Accepted answer
  
  They are not maximizing the gradient, the gradient is of the form
  \begin{equation}
  \nabla\_{\theta} J \approx \sum\_{t=0}^T G\_t \nabla\_{\theta} \log(\pi\_{\theta}(a\_t|s\_t))
  \end{equation}
  that means that when implementing it in software you can form your objective as
  \begin{equation}
  J = \sum\_{t=0}^T G\_t \log(\pi\_{\theta}(a\_t|s\_t))
  \end{equation}
  and then taking the gradient of that objective is equal to the policy gradient.
  
  > **Comments:**
  > **username_0**: I appreciate you taking the time to answer my naive question!  
  > **username_1**: @HaoHuynhNhat you should upvote and accept the answer if it has answered your question.  
  > **username_0**: @DavidIreland Yeah thanks for your reminder! this is my first question so I didn't know that!  
  
  ---
  
  