Answers.Body: <p>The answer to your first question is because the line 'update the critic by minimising the loss <span class="math-container">$L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$</span> is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.</p>
  <p>For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T13:48:34.493Z
Answers.LastActivityDate: 2020-08-06T13:48:34.493Z
Answers.UserDisplayName: David
Body: <p>I'm trying to understand the DDPG algorithm using Keras</p>
  <p>I found the <a href="https://keras.io/examples/rl/ddpg_pendulum/" rel="nofollow noreferrer">site</a> and started analyzing the code, I can't understand 2 things.</p>
  <p>The algorithm used to write the code presented on the page</p>
  <p><a href="https://i.sstatic.net/OnnPC.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OnnPC.jpg" alt="enter image description here" /></a></p>
  <p>In the algorithm image, updating the critic's network does not require gradient</p>
  <p>But the gradient is implemented in the code, why?</p>
  <pre><code>with tf.GradientTape() as tape:
      target_actions = target_actor(next_state_batch)
      y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
      critic_value = critic_model([state_batch, action_batch])
      critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))
  
  critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
  critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))
  </code></pre>
  <p>The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?</p>
  <pre><code>with tf.GradientTape() as tape:
      actions = actor_model(state_batch)
      critic_value = critic_model([state_batch, actions])
      # Used `-value` as we want to maximize the value given
      # by the critic for our actions
      actor_loss = -tf.math.reduce_mean(critic_value)
  
  actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
  actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-13T14:21:39.513Z
Comments.Text: I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of `critic_value(state_batch, actor_model(state_batch))`.
Comments.UserDisplayName: Hai Nguyen
Tags: <reinforcement-learning><keras><gradient-descent><ddpg><pseudocode>
Title: Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
UserDisplayName: Hubert
ThreadText: # Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
  
  (Asked by: username_0 on 2020-07-11)
  
  I'm trying to understand the DDPG algorithm using Keras
  
  I found the [site](https://keras.io/examples/rl/ddpg_pendulum/) and started analyzing the code, I can't understand 2 things.
  
  The algorithm used to write the code presented on the page
  
  [![enter image description here](https://i.sstatic.net/OnnPC.jpg)](https://i.sstatic.net/OnnPC.jpg)
  
  In the algorithm image, updating the critic's network does not require gradient
  
  But the gradient is implemented in the code, why?
  
  ```
  with tf.GradientTape() as tape:
      target_actions = target_actor(next_state_batch)
      y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
      critic_value = critic_model([state_batch, action_batch])
      critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))
  
  critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
  critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))
  
  ```
  
  The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?
  
  ```
  with tf.GradientTape() as tape:
      actions = actor_model(state_batch)
      critic_value = critic_model([state_batch, actions])
      # Used `-value` as we want to maximize the value given
      # by the critic for our actions
      actor_loss = -tf.math.reduce_mean(critic_value)
  
  actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
  actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))
  
  ```
  
  **Tags:** reinforcement-learning, keras, gradient-descent, ddpg, pseudocode
  
  > **Comments:**
  > **username_1**: I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of `critic_value(state_batch, actor_model(state_batch))`.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-06. Score: 2
  
  The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum\_i \left( y\_i - Q(s\_i, a\_i |\theta^Q)\right)^2$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.
  
  For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.
  
  
  ---
  
  