<p>My conclusion so far is that the Actor-Critic algorithm described by Sutton and Barto does not work.</p>
<p>First, all implementations I have seen are actually REINFORE with baseline, like <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py</a> and <a href="https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33" rel="nofollow noreferrer">https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33</a></p>
<p>Second, GAE-Lambda is a method generalizing the two:
<a href="https://i.sstatic.net/vJpH1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vJpH1.png" alt="enter image description here" /></a></p>
<p>I tried SpinningUp's Vanilla Policy Gradients with the lambda parameter set to 0, thus reproducing Sutton and Barto's Actor Critic formula. It did not learn.</p>

CC BY-SA 4.0
2023-01-09T12:38:05.723Z
2023-01-09T12:38:05.723Z
Labo
<p>I have been implementing both REINFORCE with baseline and actor-critic to solve &quot;cartpole-v1&quot;.</p>
<p>As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (<a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2020.pdf</a>):
<a href="https://i.sstatic.net/PUIVc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PUIVc.png" alt="enter image description here" /></a>
<a href="https://i.sstatic.net/n9DKA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n9DKA.png" alt="enter image description here" /></a></p>
<p>Despite the codes being super similar, REINFORCE with baseline works well and actor-critic does not. I tried adding some entropy term without success.</p>
<p>Here is the code of REINFORCE:</p>
<pre><code>#%%
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import gym

from tqdm.auto import trange

gamma = 0.99


class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(4, 32)
        self.prob = nn.Linear(32, 2)
        self.critic = nn.Linear(32, 1)

    def forward(self, state):
        hidden = F.relu(self.hidden(state))
        return (
            self.prob(hidden).flatten(),
            self.critic(hidden).flatten(),
        )


def select_action(prob):
    m = torch.distributions.Categorical(logits=prob)
    action_pt = m.sample()
    return action_pt.numpy(), m.log_prob(action_pt)


def ewma(a, alpha=0.99):
    ans = []
    acc = a[0]
    for x in a:
        acc = acc * alpha + x * (1 - alpha)
        ans.append(acc)
    return ans


#%%
policy = ActorCritic()
optimizer = optim.Adam(policy.parameters(), lr=1e-2)
eps = np.finfo(np.float32).eps.item()
# make cartpole environment
env = gym.make(&quot;CartPole-v1&quot;)

all_rewards = []


with trange(5000) as pbar:
    for i_episode in pbar:
        state, _ = env.reset()
        rewards = []
        log_probs = []
        values = []

        for t in range(500):  # Don't infinite loop while learning
            state = torch.from_numpy(state).float()
            prob, value = policy(state)
            action, log_prob = select_action(prob)
            log_probs.append(log_prob)
            state, reward, done, trunc, _ = env.step(action)
            if done or trunc:
                break
            rewards.append(np.array([reward]).astype(np.float32))
            values.append(value)
        avg_reward = np.sum(rewards, axis=0).mean()
        all_rewards.append(avg_reward)
        # format float to 2 decimal places and left align with 5 spaces
        pbar.set_description(
            f&quot;Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}&quot;
        )
        if ewma(all_rewards)[-1] &gt; env.spec.reward_threshold:
            print(&quot;Solved!&quot;)
            break
        R = 0
        policy_loss = []
        critic_loss = []
        all_R = []
        for r, log_prob, V in zip(
            reversed(rewards), reversed(log_probs), reversed(values)
        ):
            R = R * gamma + torch.from_numpy(r)
            all_R.append(R)
            A = R - V

            policy_loss.append(torch.mean(-log_prob * A.detach()))
            critic_loss.append(F.huber_loss(V, R))

        optimizer.zero_grad()
        loss = torch.stack(policy_loss).sum() + torch.stack(critic_loss).sum()
        nn.utils.clip_grad_norm_(policy.parameters(), 100)
        loss.backward()
        optimizer.step()

# %%
import matplotlib.pyplot as plt

plt.plot(ewma(all_rewards, 0.99))
</code></pre>
<p>Here is what the code of actor critic looks like:</p>
<pre><code>#%%
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import gym

from tqdm.auto import trange

gamma = 0.99


class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(4, 32)
        self.prob = nn.Linear(32, 2)
        self.critic = nn.Linear(32, 1)

    def forward(self, state):
        hidden = F.relu(self.hidden(state))
        return (
            self.prob(hidden).flatten(),
            self.critic(hidden).flatten(),
        )


def select_action(prob):
    m = torch.distributions.Categorical(logits=prob)
    action_pt = m.sample()
    return action_pt.numpy(), m.log_prob(action_pt), m.entropy()


def ewma(a, alpha=0.99):
    ans = []
    acc = a[0]
    for x in a:
        acc = acc * alpha + x * (1 - alpha)
        ans.append(acc)
    return ans


#%%
policy = ActorCritic()
optimizer = optim.Adam(policy.parameters(), lr=1e-2)
eps = np.finfo(np.float32).eps.item()
# make cartpole environment
env = gym.make(&quot;CartPole-v1&quot;)

all_rewards = []


with trange(5000) as pbar:
    for i_episode in pbar:
        state, _ = env.reset()
        rewards = []
        log_probs = []
        entropies = []
        values = []
        for t in range(500):  # Don't infinite loop while learning
            state = torch.from_numpy(state).float()
            prob, value = policy(state)
            action, log_prob, entropy = select_action(prob)
            log_probs.append(log_prob)
            entropies.append(entropy)
            state, reward, done, trunc, _ = env.step(action)
            if done or trunc:
                break
            rewards.append(np.array([reward]).astype(np.float32))
            values.append(value)
        avg_reward = np.sum(rewards, axis=0).mean()
        all_rewards.append(avg_reward)
        # format float to 2 decimal places and left align with 5 spaces
        pbar.set_description(
            f&quot;Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}&quot;
        )
        if ewma(all_rewards)[-1] &gt; env.spec.reward_threshold:
            print(&quot;Solved!&quot;)
            break
        policy_loss = []
        critic_loss = []
        next_V = torch.zeros_like(values[-1])
        for r, log_prob, V in zip(
            reversed(rewards), reversed(log_probs), reversed(values)
        ):
            target_V = torch.from_numpy(r) + gamma * next_V
            A = target_V - V

            # trick to reduce variance
            policy_loss.append(torch.mean(-log_prob * A.detach()))
            critic_loss.append(F.huber_loss(V, target_V))
            next_V = V

        optimizer.zero_grad()
        loss = (
            torch.stack(policy_loss).sum()
            + torch.stack(critic_loss).sum()
            - torch.stack(entropies).sum() * 0.01
        )
        nn.utils.clip_grad_norm_(policy.parameters(), 100)
        loss.backward()
        optimizer.step()

# %%
import matplotlib.pyplot as plt

plt.plot(ewma(all_rewards, 0.99))

# %%
</code></pre>
<p>Here is the output of the first code:</p>
<p><a href="https://i.sstatic.net/PnCuB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PnCuB.png" alt="enter image description here" /></a></p>
<p>And the second code:</p>
<p><a href="https://i.sstatic.net/03d4v.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/03d4v.png" alt="enter image description here" /></a></p>
<p>The only difference I see in the code is that in the first case, the target is <code>R * gamma + r</code> and in the second case, the target is <code>next_V * gamma + r</code>.</p>
<p>There is another possible issue, that <code>next_V</code> is not detached from the computation graph.</p>
<p>Therefore, I also tried this line: <code>critic_loss.append(F.huber_loss(V, target_V.detach()))</code>.</p>
<p>But it also did not work.</p>

<reinforcement-learning><python><policy-gradients><reinforce>
What is the problem in my implementation of actor critic?
Labo
# What is the problem in my implementation of actor critic?

(Asked by: username_0 on 2023-01-08)

I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1".
As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://incompleteideas.net/book/RLbook2020.pdf):


Despite the codes being super similar, REINFORCE with baseline works well and actor-critic does not. I tried adding some entropy term without success.
Here is the code of REINFORCE:
#%%
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import gym

from tqdm.auto import trange

gamma = 0.99


class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(4, 32)
        self.prob = nn.Linear(32, 2)
        self.critic = nn.Linear(32, 1)

    def forward(self, state):
        hidden = F.relu(self.hidden(state))
        return (
            self.prob(hidden).flatten(),
            self.critic(hidden).flatten(),
        )


def select_action(prob):
    m = torch.distributions.Categorical(logits=prob)
    action_pt = m.sample()
    return action_pt.numpy(), m.log_prob(action_pt)


def ewma(a, alpha=0.99):
    ans = []
    acc = a[0]
    for x in a:
        acc = acc * alpha + x * (1 - alpha)
        ans.append(acc)
    return ans


#%%
policy = ActorCritic()
optimizer = optim.Adam(policy.parameters(), lr=1e-2)
eps = np.finfo(np.float32).eps.item()
# make cartpole environment
env = gym.make("CartPole-v1")

all_rewards = []


with trange(5000) as pbar:
    for i_episode in pbar:
        state, _ = env.reset()
        rewards = []
        log_probs = []
        values = []

        for t in range(500):  # Don't infinite loop while learning
            state = torch.from_numpy(state).float()
            prob, value = policy(state)
            action, log_prob = select_action(prob)
            log_probs.append(log_prob)
            state, reward, done, trunc, _ = env.step(action)
            if done or trunc:
                break
            rewards.append(np.array([reward]).astype(np.float32))
            values.append(value)
        avg_reward = np.sum(rewards, axis=0).mean()
        all_rewards.append(avg_reward)
        # format float to 2 decimal places and left align with 5 spaces
        pbar.set_description(
            f"Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}"
        )
        if ewma(all_rewards)[-1] > env.spec.reward_threshold:
            print("Solved!")
            break
        R = 0
        policy_loss = []
        critic_loss = []
        all_R = []
        for r, log_prob, V in zip(
            reversed(rewards), reversed(log_probs), reversed(values)
        ):
            R = R * gamma + torch.from_numpy(r)
            all_R.append(R)
            A = R - V

            policy_loss.append(torch.mean(-log_prob * A.detach()))
            critic_loss.append(F.huber_loss(V, R))

        optimizer.zero_grad()
        loss = torch.stack(policy_loss).sum() + torch.stack(critic_loss).sum()
        nn.utils.clip_grad_norm_(policy.parameters(), 100)
        loss.backward()
        optimizer.step()

# %%
import matplotlib.pyplot as plt

plt.plot(ewma(all_rewards, 0.99))

Here is what the code of actor critic looks like:
#%%
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import gym

from tqdm.auto import trange

gamma = 0.99


class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(4, 32)
        self.prob = nn.Linear(32, 2)
        self.critic = nn.Linear(32, 1)

    def forward(self, state):
        hidden = F.relu(self.hidden(state))
        return (
            self.prob(hidden).flatten(),
            self.critic(hidden).flatten(),
        )


def select_action(prob):
    m = torch.distributions.Categorical(logits=prob)
    action_pt = m.sample()
    return action_pt.numpy(), m.log_prob(action_pt), m.entropy()


def ewma(a, alpha=0.99):
    ans = []
    acc = a[0]
    for x in a:
        acc = acc * alpha + x * (1 - alpha)
        ans.append(acc)
    return ans


#%%
policy = ActorCritic()
optimizer = optim.Adam(policy.parameters(), lr=1e-2)
eps = np.finfo(np.float32).eps.item()
# make cartpole environment
env = gym.make("CartPole-v1")

all_rewards = []


with trange(5000) as pbar:
    for i_episode in pbar:
        state, _ = env.reset()
        rewards = []
        log_probs = []
        entropies = []
        values = []
        for t in range(500):  # Don't infinite loop while learning
            state = torch.from_numpy(state).float()
            prob, value = policy(state)
            action, log_prob, entropy = select_action(prob)
            log_probs.append(log_prob)
            entropies.append(entropy)
            state, reward, done, trunc, _ = env.step(action)
            if done or trunc:
                break
            rewards.append(np.array([reward]).astype(np.float32))
            values.append(value)
        avg_reward = np.sum(rewards, axis=0).mean()
        all_rewards.append(avg_reward)
        # format float to 2 decimal places and left align with 5 spaces
        pbar.set_description(
            f"Episode {i_episode + 1} reward: {ewma(all_rewards)[-1]:.2f}"
        )
        if ewma(all_rewards)[-1] > env.spec.reward_threshold:
            print("Solved!")
            break
        policy_loss = []
        critic_loss = []
        next_V = torch.zeros_like(values[-1])
        for r, log_prob, V in zip(
            reversed(rewards), reversed(log_probs), reversed(values)
        ):
            target_V = torch.from_numpy(r) + gamma * next_V
            A = target_V - V

            # trick to reduce variance
            policy_loss.append(torch.mean(-log_prob * A.detach()))
            critic_loss.append(F.huber_loss(V, target_V))
            next_V = V

        optimizer.zero_grad()
        loss = (
            torch.stack(policy_loss).sum()
            + torch.stack(critic_loss).sum()
            - torch.stack(entropies).sum() * 0.01
        )
        nn.utils.clip_grad_norm_(policy.parameters(), 100)
        loss.backward()
        optimizer.step()

# %%
import matplotlib.pyplot as plt

plt.plot(ewma(all_rewards, 0.99))

# %%

Here is the output of the first code:

And the second code:

The only difference I see in the code is that in the first case, the target is R * gamma + r and in the second case, the target is next_V * gamma + r.
There is another possible issue, that next_V is not detached from the computation graph.
Therefore, I also tried this line: critic_loss.append(F.huber_loss(V, target_V.detach())).
But it also did not work.

**Tags:** reinforcement-learning, python, policy-gradients, reinforce


---

## 1 answers

### Answer by username_0 on 2023-01-09. Score: 1

My conclusion so far is that the Actor-Critic algorithm described by Sutton and Barto does not work.

First, all implementations I have seen are actually REINFORE with baseline, like <https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py> and <https://github.com/nikhilbarhate99/Actor-Critic-PyTorch/blob/01c833e83006be5762151a29f0719cc9c03c204d/model.py#L33>

Second, GAE-Lambda is a method generalizing the two:
[![enter image description here](https://i.sstatic.net/vJpH1.png)](https://i.sstatic.net/vJpH1.png)

I tried SpinningUp's Vanilla Policy Gradients with the lambda parameter set to 0, thus reproducing Sutton and Barto's Actor Critic formula. It did not learn.


---

