Answers.Body: <p>My conclusion so far is that the Actor-Critic algorithm described by Sutton and Barto does not work.</p>
  <p>First, all implementations I have seen are actually REINFORE with baseline, like <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py</a> and <a href="https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33" rel="nofollow noreferrer">https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33</a></p>
  <p>Second, GAE-Lambda is a method generalizing the two:
  <a href="https://i.sstatic.net/vJpH1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vJpH1.png" alt="enter image description here" /></a></p>
  <p>I tried SpinningUp's Vanilla Policy Gradients with the lambda parameter set to 0, thus reproducing Sutton and Barto's Actor Critic formula. It did not learn.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T12:38:05.723Z
Answers.LastActivityDate: 2023-01-09T12:38:05.723Z
Answers.UserDisplayName: Labo
Body: <p>I have been implementing both REINFORCE with baseline and actor-critic to solve &quot;cartpole-v1&quot;.</p>
  <p>As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (<a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2020.pdf</a>):
  <a href="https://i.sstatic.net/PUIVc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PUIVc.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/n9DKA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n9DKA.png" alt="enter image description here" /></a></p>
  <p>Despite the codes being super similar, REINFORCE with baseline works well and actor-critic does not. I tried adding some entropy term without success.</p>
  <p>Here is the code of REINFORCE:</p>
  <pre><code>#%%
  import numpy as np
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  import gym
  
  from tqdm.auto import trange
  
  gamma = 0.99
  
  
  class ActorCritic(nn.Module):
      def __init__(self):
          super().__init__()
          self.hidden = nn.Linear(4, 32)
          self.prob = nn.Linear(32, 2)
          self.critic = nn.Linear(32, 1)
  
      def forward(self, state):
          hidden = F.relu(self.hidden(state))
          return (
              self.prob(hidden).flatten(),
              self.critic(hidden).flatten(),
          )
  
  
  def select_action(prob):
      m = torch.distributions.Categorical(logits=prob)
      action_pt = m.sample()
      return action_pt.numpy(), m.log_prob(action_pt)
  
  
  def ewma(a, alpha=0.99):
      ans = []
      acc = a[0]
      for x in a:
          acc = acc * alpha + x * (1 - alpha)
          ans.append(acc)
      return ans
  
  
  #%%
  policy = ActorCritic()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  # make cartpole environment
  env = gym.make(&quot;CartPole-v1&quot;)
  
  all_rewards = []
  
  
  with trange(5000) as pbar:
      for i_episode in pbar:
          state, _ = env.reset()
          rewards = []
          log_probs = []
          values = []
  
          for t in range(500):  # Don't infinite loop while learning
              state = torch.from_numpy(state).float()
              prob, value = policy(state)
              action, log_prob = select_action(prob)
              log_probs.append(log_prob)
              state, reward, done, trunc, _ = env.step(action)
              if done or trunc:
                  break
              rewards.append(np.array([reward]).astype(np.float32))
              values.append(value)
          avg_reward = np.sum(rewards, axis=0).mean()
          all_rewards.append(avg_reward)
          # format float to 2 decimal places and left align with 5 spaces
          pbar.set_description(
              f&quot;Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}&quot;
          )
          if ewma(all_rewards)[-1] &gt; env.spec.reward_threshold:
              print(&quot;Solved!&quot;)
              break
          R = 0
          policy_loss = []
          critic_loss = []
          all_R = []
          for r, log_prob, V in zip(
              reversed(rewards), reversed(log_probs), reversed(values)
          ):
              R = R * gamma + torch.from_numpy(r)
              all_R.append(R)
              A = R - V
  
              policy_loss.append(torch.mean(-log_prob * A.detach()))
              critic_loss.append(F.huber_loss(V, R))
  
          optimizer.zero_grad()
          loss = torch.stack(policy_loss).sum() + torch.stack(critic_loss).sum()
          nn.utils.clip_grad_norm_(policy.parameters(), 100)
          loss.backward()
          optimizer.step()
  
  # %%
  import matplotlib.pyplot as plt
  
  plt.plot(ewma(all_rewards, 0.99))
  </code></pre>
  <p>Here is what the code of actor critic looks like:</p>
  <pre><code>#%%
  import numpy as np
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  import gym
  
  from tqdm.auto import trange
  
  gamma = 0.99
  
  
  class ActorCritic(nn.Module):
      def __init__(self):
          super().__init__()
          self.hidden = nn.Linear(4, 32)
          self.prob = nn.Linear(32, 2)
          self.critic = nn.Linear(32, 1)
  
      def forward(self, state):
          hidden = F.relu(self.hidden(state))
          return (
              self.prob(hidden).flatten(),
              self.critic(hidden).flatten(),
          )
  
  
  def select_action(prob):
      m = torch.distributions.Categorical(logits=prob)
      action_pt = m.sample()
      return action_pt.numpy(), m.log_prob(action_pt), m.entropy()
  
  
  def ewma(a, alpha=0.99):
      ans = []
      acc = a[0]
      for x in a:
          acc = acc * alpha + x * (1 - alpha)
          ans.append(acc)
      return ans
  
  
  #%%
  policy = ActorCritic()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  # make cartpole environment
  env = gym.make(&quot;CartPole-v1&quot;)
  
  all_rewards = []
  
  
  with trange(5000) as pbar:
      for i_episode in pbar:
          state, _ = env.reset()
          rewards = []
          log_probs = []
          entropies = []
          values = []
          for t in range(500):  # Don't infinite loop while learning
              state = torch.from_numpy(state).float()
              prob, value = policy(state)
              action, log_prob, entropy = select_action(prob)
              log_probs.append(log_prob)
              entropies.append(entropy)
              state, reward, done, trunc, _ = env.step(action)
              if done or trunc:
                  break
              rewards.append(np.array([reward]).astype(np.float32))
              values.append(value)
          avg_reward = np.sum(rewards, axis=0).mean()
          all_rewards.append(avg_reward)
          # format float to 2 decimal places and left align with 5 spaces
          pbar.set_description(
              f&quot;Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}&quot;
          )
          if ewma(all_rewards)[-1] &gt; env.spec.reward_threshold:
              print(&quot;Solved!&quot;)
              break
          policy_loss = []
          critic_loss = []
          next_V = torch.zeros_like(values[-1])
          for r, log_prob, V in zip(
              reversed(rewards), reversed(log_probs), reversed(values)
          ):
              target_V = torch.from_numpy(r) + gamma * next_V
              A = target_V - V
  
              # trick to reduce variance
              policy_loss.append(torch.mean(-log_prob * A.detach()))
              critic_loss.append(F.huber_loss(V, target_V))
              next_V = V
  
          optimizer.zero_grad()
          loss = (
              torch.stack(policy_loss).sum()
              + torch.stack(critic_loss).sum()
              - torch.stack(entropies).sum() * 0.01
          )
          nn.utils.clip_grad_norm_(policy.parameters(), 100)
          loss.backward()
          optimizer.step()
  
  # %%
  import matplotlib.pyplot as plt
  
  plt.plot(ewma(all_rewards, 0.99))
  
  # %%
  </code></pre>
  <p>Here is the output of the first code:</p>
  <p><a href="https://i.sstatic.net/PnCuB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PnCuB.png" alt="enter image description here" /></a></p>
  <p>And the second code:</p>
  <p><a href="https://i.sstatic.net/03d4v.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/03d4v.png" alt="enter image description here" /></a></p>
  <p>The only difference I see in the code is that in the first case, the target is <code>R * gamma + r</code> and in the second case, the target is <code>next_V * gamma + r</code>.</p>
  <p>There is another possible issue, that <code>next_V</code> is not detached from the computation graph.</p>
  <p>Therefore, I also tried this line: <code>critic_loss.append(F.huber_loss(V, target_V.detach()))</code>.</p>
  <p>But it also did not work.</p>
  
Tags: <reinforcement-learning><python><policy-gradients><reinforce>
Title: What is the problem in my implementation of actor critic?
UserDisplayName: Labo
ThreadText: # What is the problem in my implementation of actor critic?
  
  (Asked by: username_0 on 2023-01-08)
  
  I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1".
  
  As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (<http://incompleteideas.net/book/RLbook2020.pdf>):
  [![enter image description here](https://i.sstatic.net/PUIVc.png)](https://i.sstatic.net/PUIVc.png)
  [![enter image description here](https://i.sstatic.net/n9DKA.png)](https://i.sstatic.net/n9DKA.png)
  
  Despite the codes being super similar, REINFORCE with baseline works well and actor-critic does not. I tried adding some entropy term without success.
  
  Here is the code of REINFORCE:
  
  ```
  #%%
  import numpy as np
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  import gym
  
  from tqdm.auto import trange
  
  gamma = 0.99
  
  class ActorCritic(nn.Module):
      def __init__(self):
          super().__init__()
          self.hidden = nn.Linear(4, 32)
          self.prob = nn.Linear(32, 2)
          self.critic = nn.Linear(32, 1)
  
      def forward(self, state):
          hidden = F.relu(self.hidden(state))
          return (
              self.prob(hidden).flatten(),
              self.critic(hidden).flatten(),
          )
  
  def select_action(prob):
      m = torch.distributions.Categorical(logits=prob)
      action_pt = m.sample()
      return action_pt.numpy(), m.log_prob(action_pt)
  
  def ewma(a, alpha=0.99):
      ans = []
      acc = a[0]
      for x in a:
          acc = acc * alpha + x * (1 - alpha)
          ans.append(acc)
      return ans
  
  #%%
  policy = ActorCritic()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  # make cartpole environment
  env = gym.make("CartPole-v1")
  
  all_rewards = []
  
  with trange(5000) as pbar:
      for i_episode in pbar:
          state, _ = env.reset()
          rewards = []
          log_probs = []
          values = []
  
          for t in range(500):  # Don't infinite loop while learning
              state = torch.from_numpy(state).float()
              prob, value = policy(state)
              action, log_prob = select_action(prob)
              log_probs.append(log_prob)
              state, reward, done, trunc, _ = env.step(action)
              if done or trunc:
                  break
              rewards.append(np.array([reward]).astype(np.float32))
              values.append(value)
          avg_reward = np.sum(rewards, axis=0).mean()
          all_rewards.append(avg_reward)
          # format float to 2 decimal places and left align with 5 spaces
          pbar.set_description(
              f"Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}"
          )
          if ewma(all_rewards)[-1] > env.spec.reward_threshold:
              print("Solved!")
              break
          R = 0
          policy_loss = []
          critic_loss = []
          all_R = []
          for r, log_prob, V in zip(
              reversed(rewards), reversed(log_probs), reversed(values)
          ):
              R = R * gamma + torch.from_numpy(r)
              all_R.append(R)
              A = R - V
  
              policy_loss.append(torch.mean(-log_prob * A.detach()))
              critic_loss.append(F.huber_loss(V, R))
  
          optimizer.zero_grad()
          loss = torch.stack(policy_loss).sum() + torch.stack(critic_loss).sum()
          nn.utils.clip_grad_norm_(policy.parameters(), 100)
          loss.backward()
          optimizer.step()
  
  # %%
  import matplotlib.pyplot as plt
  
  plt.plot(ewma(all_rewards, 0.99))
  
  ```
  
  Here is what the code of actor critic looks like:
  
  ```
  #%%
  import numpy as np
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  import gym
  
  from tqdm.auto import trange
  
  gamma = 0.99
  
  class ActorCritic(nn.Module):
      def __init__(self):
          super().__init__()
          self.hidden = nn.Linear(4, 32)
          self.prob = nn.Linear(32, 2)
          self.critic = nn.Linear(32, 1)
  
      def forward(self, state):
          hidden = F.relu(self.hidden(state))
          return (
              self.prob(hidden).flatten(),
              self.critic(hidden).flatten(),
          )
  
  def select_action(prob):
      m = torch.distributions.Categorical(logits=prob)
      action_pt = m.sample()
      return action_pt.numpy(), m.log_prob(action_pt), m.entropy()
  
  def ewma(a, alpha=0.99):
      ans = []
      acc = a[0]
      for x in a:
          acc = acc * alpha + x * (1 - alpha)
          ans.append(acc)
      return ans
  
  #%%
  policy = ActorCritic()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  # make cartpole environment
  env = gym.make("CartPole-v1")
  
  all_rewards = []
  
  with trange(5000) as pbar:
      for i_episode in pbar:
          state, _ = env.reset()
          rewards = []
          log_probs = []
          entropies = []
          values = []
          for t in range(500):  # Don't infinite loop while learning
              state = torch.from_numpy(state).float()
              prob, value = policy(state)
              action, log_prob, entropy = select_action(prob)
              log_probs.append(log_prob)
              entropies.append(entropy)
              state, reward, done, trunc, _ = env.step(action)
              if done or trunc:
                  break
              rewards.append(np.array([reward]).astype(np.float32))
              values.append(value)
          avg_reward = np.sum(rewards, axis=0).mean()
          all_rewards.append(avg_reward)
          # format float to 2 decimal places and left align with 5 spaces
          pbar.set_description(
              f"Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}"
          )
          if ewma(all_rewards)[-1] > env.spec.reward_threshold:
              print("Solved!")
              break
          policy_loss = []
          critic_loss = []
          next_V = torch.zeros_like(values[-1])
          for r, log_prob, V in zip(
              reversed(rewards), reversed(log_probs), reversed(values)
          ):
              target_V = torch.from_numpy(r) + gamma * next_V
              A = target_V - V
  
              # trick to reduce variance
              policy_loss.append(torch.mean(-log_prob * A.detach()))
              critic_loss.append(F.huber_loss(V, target_V))
              next_V = V
  
          optimizer.zero_grad()
          loss = (
              torch.stack(policy_loss).sum()
              + torch.stack(critic_loss).sum()
              - torch.stack(entropies).sum() * 0.01
          )
          nn.utils.clip_grad_norm_(policy.parameters(), 100)
          loss.backward()
          optimizer.step()
  
  # %%
  import matplotlib.pyplot as plt
  
  plt.plot(ewma(all_rewards, 0.99))
  
  # %%
  
  ```
  
  Here is the output of the first code:
  
  [![enter image description here](https://i.sstatic.net/PnCuB.png)](https://i.sstatic.net/PnCuB.png)
  
  And the second code:
  
  [![enter image description here](https://i.sstatic.net/03d4v.png)](https://i.sstatic.net/03d4v.png)
  
  The only difference I see in the code is that in the first case, the target is `R * gamma + r` and in the second case, the target is `next_V * gamma + r`.
  
  There is another possible issue, that `next_V` is not detached from the computation graph.
  
  Therefore, I also tried this line: `critic_loss.append(F.huber_loss(V, target_V.detach()))`.
  
  But it also did not work.
  
  **Tags:** reinforcement-learning, python, policy-gradients, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-01-09. Score: 1
  
  My conclusion so far is that the Actor-Critic algorithm described by Sutton and Barto does not work.
  
  First, all implementations I have seen are actually REINFORE with baseline, like <https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py> and <https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33>
  
  Second, GAE-Lambda is a method generalizing the two:
  [![enter image description here](https://i.sstatic.net/vJpH1.png)](https://i.sstatic.net/vJpH1.png)
  
  I tried SpinningUp's Vanilla Policy Gradients with the lambda parameter set to 0, thus reproducing Sutton and Barto's Actor Critic formula. It did not learn.
  
  
  ---
  
  