Answers.Body: <p>Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T15:50:58.787Z
Answers.Text: So to have those entries I have to compute the jacobian?
Answers.UserDisplayName: unter_983
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T18:32:07.713Z
Answers.Text: Yeah, basically.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T15:13:49.083Z
Answers.LastActivityDate: 2020-06-21T15:13:49.083Z
Answers.UserDisplayName: harwiltz
Body: <p>I'm trying to understand the DDPG algorithm shown at this <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">page</a>. I don't know what should the result of the gradient at step 14 be.</p>
  <p><a href="https://i.sstatic.net/vPmro.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vPmro.png" alt="enter image description here" /></a></p>
  <p>Is it a scalar that I have to use to update all the weights (so all weights are updated with the same value)? Or is it a list with a different values to use for updating for each weight? I'm used to working with loss functions and an <span class="math-container">$y$</span> target, but here I don't have them so I'm quite confused.</p>
  
Tags: <reinforcement-learning><policy-gradients><actor-critic-methods><ddpg>
Title: In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
UserDisplayName: unter_983
ThreadText: # In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
  
  (Asked by: username_0 on 2020-06-21)
  
  I'm trying to understand the DDPG algorithm shown at this [page](https://spinningup.openai.com/en/latest/algorithms/ddpg.html). I don't know what should the result of the gradient at step 14 be.
  
  [![enter image description here](https://i.sstatic.net/vPmro.png)](https://i.sstatic.net/vPmro.png)
  
  Is it a scalar that I have to use to update all the weights (so all weights are updated with the same value)? Or is it a list with a different values to use for updating for each weight? I'm used to working with loss functions and an $y$ target, but here I don't have them so I'm quite confused.
  
  **Tags:** reinforcement-learning, policy-gradients, actor-critic-methods, ddpg
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-21. Score: 1
  
  Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.
  
  > **Comments:**
  > **username_0**: So to have those entries I have to compute the jacobian?  
  > **username_1**: Yeah, basically.  
  
  ---
  
  