Answers.Body: <p>I'm pretty sure you should throw a minus sign in front of the selected log probability in your loss function. (this turned out to be wrong and the code works fine: <a href="https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin" rel="nofollow noreferrer">https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin</a>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-07T16:55:29.437Z
Answers.Text: Doesn't the minus sign in front of `weighted.sum()` a couple of lines below have the same effect?
  I tried that and the output was the same. The loss was decreasing during training, but the agent was perfoming worse every time. It really puzzles me
Answers.UserDisplayName: wildBass
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T11:10:23.713Z
Answers.Text: yeah you're right. 
  ive tried to run your code just now and I dont get the same result
Answers.UserDisplayName: Leonhard Piff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T11:41:45.707Z
Answers.Text: I have a link to a colab where I ran the code it worked and I turned a copy of it into A2C.
  
  [link](https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharing)
  https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin
Answers.UserDisplayName: Leonhard Piff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T15:53:36.187Z
Answers.Text: Wow, thanks for taking the time Leonhard!
  
  Yeah, this morning I did those changes, and I also changed the reward for -1 if the episode is terminated, and 0 otherwise, and worked perfectly. 
  
  I still don't know exactly what was happening before. I was really confused by having a decreasing loss and the agent performing worse and worse each time. 
  But anyway, it works now. 
  
  And thanks for the A2C implementation! I'm studying RL at the moment and that will help me!
Answers.UserDisplayName: wildBass
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T22:44:18.590Z
Answers.Text: No worries show me the next step you take please
Answers.UserDisplayName: Leonhard Piff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-07T15:57:21.807Z
Answers.LastActivityDate: 2024-11-08T11:42:52.613Z
Answers.LastEditDate: 2024-11-08T11:42:52.613Z
Answers.UserDisplayName: Leonhard Piff
Body: <p>I'm implementing a basic Vanilla Policy Gradient algorithm for the CartPole-v1 gymnasium environment, and I don't know what I'm doing wrong. No matter what I try, during the training loop the loss decreases (so the model is actually learning something), but the episode total reward also decreases until it reaches around 9-10 steps (which I imagine is the minimum number of steps needed to make the pole fall). So it's learning to do it bad!</p>
  <p>In this algorithm there are a few moving parts that influence each other, and I run out of ideas to see where the problem could be.</p>
  <p>For the discounted rewards I'm using</p>
  <p><span class="math-container">$ Q_{k,t} = \sum_{i=0}{\gamma^{i-t} r_i} $</span></p>
  <p>for <span class="math-container">$k$</span> = all episodes and <span class="math-container">$t$</span> = all steps in an episode.
  And for the loss:</p>
  <p><span class="math-container">$ L = -\sum_{k,t}Q_{k,t}log\pi_{\theta}(a_t | s_t)$</span></p>
  <p>The code is a mix from Maxim Lapan's Deep RL Hands-On book, Karpathy's Pong example (blog, code), and personal tweaks.</p>
  <p>Here's my code:</p>
  <pre><code>import gymnasium as gym
  import torch
  from torch import nn
  import torch.nn.functional as F
  from torch.nn.init import xavier_uniform_
  import numpy as np
  
  GAMMA = 0.99
  LEARNING_RATE = 0.001
  BATCH_SIZE = 4
  DEVICE = torch.device('mps')
  
  
  class XavierLinear(nn.Linear):
      def __init__(self, in_features: int, out_features: int, bias: bool = True, device=None, dtype=None) -&gt; None:
          super().__init__(in_features, out_features, bias, device, dtype)
          xavier_uniform_(self.weight)
  
  
  class VPG(nn.Module):
      def __init__(self, input_size, output_size):
          super(VPG, self).__init__()
          self.net = nn.Sequential(
              XavierLinear(input_size, 128),
              nn.ReLU(),
              XavierLinear(128, output_size), 
          )
  
      def forward(self, x):
          return F.softmax(self.net(x), dim=0)
  
  
  def run_episode(model, env):
      obs = env.reset()[0]
      obs = torch.Tensor(env.reset()[0]).to(DEVICE)
      te = tr = False
      rewards, outputs, actions = [], [], []
      while not (te or tr):
          probs = model(obs)
          action = probs.multinomial(1).item()
          obs, r, te, tr, _ = env.step(action)
          obs = torch.Tensor(obs).to(DEVICE)
          if (te or tr):
              r = 0 # Here I also tried -1 if the pole falls and 0 otherwise
          rewards.append(r)
          outputs.append(probs)
          actions.append(action)
      return torch.Tensor(rewards).to(DEVICE), torch.concatenate(outputs).reshape(len(rewards), 2), actions
  
  def discount_rewards(rewards):
      discounted_r = torch.zeros_like(rewards)
      additive_r = 0
      for idx in range(len(rewards)-1, -1, -1):
          to_add = GAMMA * additive_r
          additive_r = to_add + rewards[idx]
          discounted_r[idx] = additive_r
      return discounted_r.to(DEVICE)
  
  def loss_function(discounted_r, probs, actions):
      logprobs = torch.log(probs)
      selected = logprobs[range(probs.shape[0]), actions]
      # discounted_r = (discounted_r - discounted_r.mean()) / discounted_r.std()
      weighted = selected * discounted_r
      return -weighted.sum()
  
  # The actual training loop:
  
  episode_total_reward = 0
  batch_losses = torch.Tensor().to(DEVICE)
  batch_actions = []
  batch_disc_r = torch.Tensor().to(DEVICE)
  batch_probs = torch.Tensor().to(DEVICE)
  best_ep_reward = 0
  losses, ep_total_lenghts = [], [0]
  
  episodes = 0
  TARGET_REWARD = 100
  
  env = gym.make(&quot;CartPole-v1&quot;)
  model = VPG(env.observation_space.shape[0],
              2).to(DEVICE)
  optim = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
  
  while np.array(ep_total_lenghts)[-100:].mean() &lt; TARGET_REWARD:
      rewards, probs, actions = run_episode(model, env)
      discounted_r = discount_rewards(rewards)
      episode_total_reward = rewards.shape[0]
      ep_total_lenghts.append(episode_total_reward)
      episodes += 1
      batch_actions += actions
      batch_disc_r = torch.concatenate([batch_disc_r, discounted_r])    
      batch_probs = torch.concatenate([batch_probs, probs])    
  
      if episodes % BATCH_SIZE == 0:
          loss = loss_function(batch_disc_r, batch_probs, batch_actions)
          losses.append(loss.item())
          model.zero_grad()
          loss.backward()
          optim.step()
          batch_actions = []
          batch_disc_r = torch.Tensor().to(DEVICE)
          batch_probs = torch.Tensor().to(DEVICE)
          print(f&quot;Episode {episodes}. Loss: {loss}. Reward: {episode_total_reward}&quot;)
  print(f&quot;Success in {episodes} episodes. Loss: {loss}. Reward: {episode_total_reward}&quot;)
  </code></pre>
  <p>Tried: changing signs in loss functions, changing rewards (non-terminal step = 0 and terminal step = -1), updating manually the weights (adding the gradient or substracting it instead of using Adam optimizer). In each case I get the same: the loss decreases but the agent doens't learn to keep the pole up. Actually, it learns to throw it.</p>
  <p>Expectation: Loss decreases and episode total reward (steps played) increases.</p>
  
Tags: <reinforcement-learning><python><policy-gradients>
Title: Deep RL problem: Loss decreases but agent doesn't learn
UserDisplayName: wildBass
ThreadText: # Deep RL problem: Loss decreases but agent doesn't learn
  
  (Asked by: username_0 on 2024-11-07)
  
  I'm implementing a basic Vanilla Policy Gradient algorithm for the CartPole-v1 gymnasium environment, and I don't know what I'm doing wrong. No matter what I try, during the training loop the loss decreases (so the model is actually learning something), but the episode total reward also decreases until it reaches around 9-10 steps (which I imagine is the minimum number of steps needed to make the pole fall). So it's learning to do it bad!
  
  In this algorithm there are a few moving parts that influence each other, and I run out of ideas to see where the problem could be.
  
  For the discounted rewards I'm using
  
  $ Q\_{k,t} = \sum\_{i=0}{\gamma^{i-t} r\_i} $
  
  for $k$ = all episodes and $t$ = all steps in an episode.
  And for the loss:
  
  $ L = -\sum\_{k,t}Q\_{k,t}log\pi\_{\theta}(a\_t | s\_t)$
  
  The code is a mix from Maxim Lapan's Deep RL Hands-On book, Karpathy's Pong example (blog, code), and personal tweaks.
  
  Here's my code:
  
  ```
  import gymnasium as gym
  import torch
  from torch import nn
  import torch.nn.functional as F
  from torch.nn.init import xavier_uniform_
  import numpy as np
  
  GAMMA = 0.99
  LEARNING_RATE = 0.001
  BATCH_SIZE = 4
  DEVICE = torch.device('mps')
  
  class XavierLinear(nn.Linear):
      def __init__(self, in_features: int, out_features: int, bias: bool = True, device=None, dtype=None) -> None:
          super().__init__(in_features, out_features, bias, device, dtype)
          xavier_uniform_(self.weight)
  
  class VPG(nn.Module):
      def __init__(self, input_size, output_size):
          super(VPG, self).__init__()
          self.net = nn.Sequential(
              XavierLinear(input_size, 128),
              nn.ReLU(),
              XavierLinear(128, output_size), 
          )
  
      def forward(self, x):
          return F.softmax(self.net(x), dim=0)
  
  def run_episode(model, env):
      obs = env.reset()[0]
      obs = torch.Tensor(env.reset()[0]).to(DEVICE)
      te = tr = False
      rewards, outputs, actions = [], [], []
      while not (te or tr):
          probs = model(obs)
          action = probs.multinomial(1).item()
          obs, r, te, tr, _ = env.step(action)
          obs = torch.Tensor(obs).to(DEVICE)
          if (te or tr):
              r = 0 # Here I also tried -1 if the pole falls and 0 otherwise
          rewards.append(r)
          outputs.append(probs)
          actions.append(action)
      return torch.Tensor(rewards).to(DEVICE), torch.concatenate(outputs).reshape(len(rewards), 2), actions
  
  def discount_rewards(rewards):
      discounted_r = torch.zeros_like(rewards)
      additive_r = 0
      for idx in range(len(rewards)-1, -1, -1):
          to_add = GAMMA * additive_r
          additive_r = to_add + rewards[idx]
          discounted_r[idx] = additive_r
      return discounted_r.to(DEVICE)
  
  def loss_function(discounted_r, probs, actions):
      logprobs = torch.log(probs)
      selected = logprobs[range(probs.shape[0]), actions]
      # discounted_r = (discounted_r - discounted_r.mean()) / discounted_r.std()
      weighted = selected * discounted_r
      return -weighted.sum()
  
  # The actual training loop:
  
  episode_total_reward = 0
  batch_losses = torch.Tensor().to(DEVICE)
  batch_actions = []
  batch_disc_r = torch.Tensor().to(DEVICE)
  batch_probs = torch.Tensor().to(DEVICE)
  best_ep_reward = 0
  losses, ep_total_lenghts = [], [0]
  
  episodes = 0
  TARGET_REWARD = 100
  
  env = gym.make("CartPole-v1")
  model = VPG(env.observation_space.shape[0],
              2).to(DEVICE)
  optim = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
  
  while np.array(ep_total_lenghts)[-100:].mean() < TARGET_REWARD:
      rewards, probs, actions = run_episode(model, env)
      discounted_r = discount_rewards(rewards)
      episode_total_reward = rewards.shape[0]
      ep_total_lenghts.append(episode_total_reward)
      episodes += 1
      batch_actions += actions
      batch_disc_r = torch.concatenate([batch_disc_r, discounted_r])    
      batch_probs = torch.concatenate([batch_probs, probs])    
  
      if episodes % BATCH_SIZE == 0:
          loss = loss_function(batch_disc_r, batch_probs, batch_actions)
          losses.append(loss.item())
          model.zero_grad()
          loss.backward()
          optim.step()
          batch_actions = []
          batch_disc_r = torch.Tensor().to(DEVICE)
          batch_probs = torch.Tensor().to(DEVICE)
          print(f"Episode {episodes}. Loss: {loss}. Reward: {episode_total_reward}")
  print(f"Success in {episodes} episodes. Loss: {loss}. Reward: {episode_total_reward}")
  
  ```
  
  Tried: changing signs in loss functions, changing rewards (non-terminal step = 0 and terminal step = -1), updating manually the weights (adding the gradient or substracting it instead of using Adam optimizer). In each case I get the same: the loss decreases but the agent doens't learn to keep the pole up. Actually, it learns to throw it.
  
  Expectation: Loss decreases and episode total reward (steps played) increases.
  
  **Tags:** reinforcement-learning, python, policy-gradients
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-07. Score: 0
  
  I'm pretty sure you should throw a minus sign in front of the selected log probability in your loss function. (this turned out to be wrong and the code works fine: <https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin>)
  
  > **Comments:**
  > **username_0**: Doesn't the minus sign in front of `weighted.sum()` a couple of lines below have the same effect? I tried that and the output was the same. The loss was decreasing during training, but the agent was perfoming worse every time. It really puzzles me  
  > **username_1**: yeah you're right.  ive tried to run your code just now and I dont get the same result  
  > **username_1**: I have a link to a colab where I ran the code it worked and I turned a copy of it into A2C.  [link](https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharing) https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin  
  > **username_0**: Wow, thanks for taking the time username_1!  Yeah, this morning I did those changes, and I also changed the reward for -1 if the episode is terminated, and 0 otherwise, and worked perfectly.   I still don't know exactly what was happening before. I was really confused by having a decreasing loss and the agent performing worse and worse each time.  But anyway, it works now.   And thanks for the A2C implementation! I'm studying RL at the moment and that will help me!  
  > **username_1**: No worries show me the next step you take please  
  
  ---
  
  