for (log_prob, value), r in zip(saved_actions, rewards):
        reward = r - value.data[0]
        policy_losses.append(-log_prob * reward)
        value_losses.append(F.smooth_l1_loss(value, Variable(torch.Tensor([r]))))

So in the current code, if you see how the loss is calculated, you’d see that the critic is taught to learn the reward of a particular state, action pair, given by r, or the action-value function rather than the advantage function. In A2C, the critic would learn the advantage function.