Answers.Body: <p>The usual assumption in deep learning is that you want to <em>minimize</em> a loss function, say <span class="math-container">$\mathcal L(\theta)$</span>. Follows that the SGD update rule is
  <span class="math-container">$$\theta \leftarrow \theta - \eta\nabla_\theta\mathcal L(\theta),$$</span>
  where <span class="math-container">$\eta$</span> is the learning rate. As you can notice we usually <em>subtract</em> the gradient to the parameters in order to minimize: since we take the negative direction. In the Sutton &amp; Barto's book they assume a maximization: in fact there is a &quot;+&quot; sign in the weights update rule, and so you don't have to change the loss fn.</p>
  <p>Said that, when the optimizer's default assumption is to minimize, you multiply the loss by <span class="math-container">$-1$</span> to maximize, i.e. to perform gradient ascent. Indeed, this would flip the gradient's direction which is then flipped again when taking the negative direction (i.e., <span class="math-container">$-\eta$</span>) in the update rule.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-23T16:07:21.280Z
Answers.Text: Great answer, thanks! Follow up question: the update for **w** in the pseudocode also seems to be gradient ascent then, since it has a "+", but the guide *doesn't* multiply critic loss by -1; why is this?
Answers.UserDisplayName: jasooney23
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T13:50:53.990Z
Answers.Text: @jasooney23 The critic loss is usually a minimization of a squared difference, like $\min_w (G-V_w)^2$: this is shown in equation 9.4 of the book (chapter 9.3 - page 201). Now, if you look at eq 9.5 (which is the one in the algo) you notice a rearrangement of the terms where $+\alpha$ appears. I think this is because they multiply by the *negative* derivative of the loss, which is $-2 (v_\pi-\hat v)$. But in practice you don't need to do this since autodiff handles that, you only need to care about the sign.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-23T07:51:26.523Z
Answers.LastActivityDate: 2023-08-23T07:51:26.523Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>Reading through the <a href="https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic" rel="nofollow noreferrer">TensorFlow guide for Actor-Critic learning</a>, I saw that the actor loss is multiplied by -1 when calculating:</p>
  <p><a href="https://i.sstatic.net/B0JaY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B0JaY.png" alt="2" /></a></p>
  <p>The guide says this is to maximize the probabilities of actions with high rewards by minimizing its loss. My understanding may be flawed, but by taking multiplying the loss by -1, wouldn't the end gradients also be multiplied by -1 (and hence flipped), instead <strong>increasing</strong> the loss function?</p>
  <p>Additionally, in <em>Reinforcement Learning: An Introduction</em> (Sutton and Barto), you see this instead, without the negative sign:</p>
  <p><a href="https://i.sstatic.net/VL4dn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VL4dn.png" alt="3" /></a></p>
  <p>Specifically, this update here for the actor, where the gradient is <strong>not</strong> multiplied by -1:</p>
  <p><a href="https://i.sstatic.net/KOyjk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KOyjk.png" alt="4" /></a></p>
  <p>Is there anything different between these two updates? I feel like there's an important concept or piece of calculus I might be missing here. Thanks!</p>
  
Tags: <reinforcement-learning><actor-critic-methods>
Title: Reinforcement Learning - Why is actor loss in Actor-Critic multiplied by -1?
UserDisplayName: jasooney23
ThreadText: # Reinforcement Learning - Why is actor loss in Actor-Critic multiplied by -1?
  
  (Asked by: username_0 on 2023-08-23)
  
  Reading through the [TensorFlow guide for Actor-Critic learning](https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic), I saw that the actor loss is multiplied by -1 when calculating:
  
  [![2](https://i.sstatic.net/B0JaY.png)](https://i.sstatic.net/B0JaY.png)
  
  The guide says this is to maximize the probabilities of actions with high rewards by minimizing its loss. My understanding may be flawed, but by taking multiplying the loss by -1, wouldn't the end gradients also be multiplied by -1 (and hence flipped), instead **increasing** the loss function?
  
  Additionally, in *Reinforcement Learning: An Introduction* (Sutton and Barto), you see this instead, without the negative sign:
  
  [![3](https://i.sstatic.net/VL4dn.png)](https://i.sstatic.net/VL4dn.png)
  
  Specifically, this update here for the actor, where the gradient is **not** multiplied by -1:
  
  [![4](https://i.sstatic.net/KOyjk.png)](https://i.sstatic.net/KOyjk.png)
  
  Is there anything different between these two updates? I feel like there's an important concept or piece of calculus I might be missing here. Thanks!
  
  **Tags:** reinforcement-learning, actor-critic-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-23. Score: 1 | Accepted answer
  
  The usual assumption in deep learning is that you want to *minimize* a loss function, say $\mathcal L(\theta)$. Follows that the SGD update rule is
  $$\theta \leftarrow \theta - \eta\nabla\_\theta\mathcal L(\theta),$$
  where $\eta$ is the learning rate. As you can notice we usually *subtract* the gradient to the parameters in order to minimize: since we take the negative direction. In the Sutton & Barto's book they assume a maximization: in fact there is a "+" sign in the weights update rule, and so you don't have to change the loss fn.
  
  Said that, when the optimizer's default assumption is to minimize, you multiply the loss by $-1$ to maximize, i.e. to perform gradient ascent. Indeed, this would flip the gradient's direction which is then flipped again when taking the negative direction (i.e., $-\eta$) in the update rule.
  
  > **Comments:**
  > **username_0**: Great answer, thanks! Follow up question: the update for **w** in the pseudocode also seems to be gradient ascent then, since it has a "+", but the guide *doesn't* multiply critic loss by -1; why is this?  
  > **username_1**: @username_0 The critic loss is usually a minimization of a squared difference, like $\min_w (G-V_w)^2$: this is shown in equation 9.4 of the book (chapter 9.3 - page 201). Now, if you look at eq 9.5 (which is the one in the algo) you notice a rearrangement of the terms where $+\alpha$ appears. I think this is because they multiply by the *negative* derivative of the loss, which is $-2 (v_\pi-\hat v)$. But in practice you don't need to do this since autodiff handles that, you only need to care about the sign.  
  
  ---
  
  