Answers.Body: <p>The answer to your first question is because the line 'update the critic by minimising the loss <span class="math-container">$L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$</span> is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.</p>
  <p>For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T13:48:34.493Z
Answers.LastActivityDate: 2020-08-06T13:48:34.493Z
Answers.UserDisplayName: David
Body: <p>I'm trying to understand the DDPG algorithm using Keras</p>
  <p>I found the <a href="https://keras.io/examples/rl/ddpg_pendulum/" rel="nofollow noreferrer">site</a> and started analyzing the code, I can't understand 2 things.</p>
  <p>The algorithm used to write the code presented on the page</p>
  <p><a href="https://i.sstatic.net/OnnPC.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OnnPC.jpg" alt="enter image description here" /></a></p>
  <p>In the algorithm image, updating the critic's network does not require gradient</p>
  <p>But the gradient is implemented in the code, why?</p>
  <pre><code>with tf.GradientTape() as tape:
      target_actions = target_actor(next_state_batch)
      y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
      critic_value = critic_model([state_batch, action_batch])
      critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))
  
  critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
  critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))
  </code></pre>
  <p>The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?</p>
  <pre><code>with tf.GradientTape() as tape:
      actions = actor_model(state_batch)
      critic_value = critic_model([state_batch, actions])
      # Used `-value` as we want to maximize the value given
      # by the critic for our actions
      actor_loss = -tf.math.reduce_mean(critic_value)
  
  actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
  actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-13T14:21:39.513Z
Comments.Text: I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of `critic_value(state_batch, actor_model(state_batch))`.
Comments.UserDisplayName: Hai Nguyen
Tags: <reinforcement-learning><keras><gradient-descent><ddpg><pseudocode>
Title: Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
UserDisplayName: Hubert
ThreadText: # Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
  
  (Asked by: username_0 on 2020-07-11)
  
  I'm trying to understand the DDPG algorithm using Keras
  I found the site and started analyzing the code, I can't understand 2 things.
  The algorithm used to write the code presented on the page
  
  In the algorithm image, updating the critic's network does not require gradient
  But the gradient is implemented in the code, why?
  with tf.GradientTape() as tape:
      target_actions = target_actor(next_state_batch)
      y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
      critic_value = critic_model([state_batch, action_batch])
      critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))
  
  critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
  critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))
  
  The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?
  with tf.GradientTape() as tape:
      actions = actor_model(state_batch)
      critic_value = critic_model([state_batch, actions])
      # Used `-value` as we want to maximize the value given
      # by the critic for our actions
      actor_loss = -tf.math.reduce_mean(critic_value)
  
  actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
  actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))
  
  **Tags:** reinforcement-learning, keras, gradient-descent, ddpg, pseudocode
  > **Comments:**
  > **username_1**: I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of `critic_value(state_batch, actor_model(state_batch))`.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-06. Score: 2
  
  The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum\_i \left( y\_i - Q(s\_i, a\_i |\theta^Q)\right)^2$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.
  
  For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.
  
  
  ---
  
  