Answers.Body: <p>In the automatic differentiation procedure after <strong>backward</strong> pass
  the gradient with respect to the scalar is added to the current gradient. Without calling <strong>zero_grad</strong> you will have the sum of all gradients, calcluated before, with the current gradient.</p>
  <p>Therefore, <code>optimizer.step()</code> will do not this:</p>
  <pre><code>w = w - eta * grad L[i] # L[i] - loss function for the i-th sample
  </code></pre>
  <p>But rather:</p>
  <pre><code>w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples
  </code></pre>
  <p>Which is not the desired behavior.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T12:09:02.730Z
Answers.Text: Worth noting that this is an implementation detail in PyTorch, and other frameworks may do the same implicitly. It is a nice feature when there's more than one source of gradients, but for simple NNs it's just some boilerplate code that needs to be written by the framework user
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T23:07:01.847Z
Answers.Text: So, if we don't use net.`zero_grad()` then the updation of weights will go wrong because of issue in updation of gradients. Updation of weight takes the sum of all the gradients calculated so far instead of the current gradient.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-19T18:22:56.767Z
Answers.Text: @hanugm, yes, precisely, it can be beneficial to use gradient information from the previous timesteps, like in the momentum optimizers - Nesterov, Adam, but not in tihs way. With the decay of the importance of gradients from previous timestamps with time.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T05:09:11.183Z
Answers.LastActivityDate: 2021-07-18T05:09:11.183Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>I agree with the answer from @spiridon_the_sun_rotator .</p>
  <p>If you don't zerorize the gradient, before calling loss.backward()
  then gradient will be accumulated which is not what you want. you want to calculate the gradient associated with the current step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T16:13:20.103Z
Answers.Text: That answer has already been given.
Answers.UserDisplayName: Chenmunka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T14:30:14.280Z
Answers.LastActivityDate: 2024-07-09T14:30:14.280Z
Answers.UserDisplayName: saleh
Body: <p>Consider the <a href="https://pytorch.org/docs/stable/notes/modules.html" rel="nofollow noreferrer">following</a> PyTorch code</p>
  <pre><code># Run a sample training loop that &quot;teaches&quot; the network
  # to output the constant zero function
  for _ in range(10000):
    input = torch.randn(4)
    output = net(input)
    loss = torch.abs(output)
    net.zero_grad()
    loss.backward()
    optimizer.step()
  </code></pre>
  <p>and its corresponding explanation on training a neural network</p>
  <p>A training loop…</p>
  <ol>
  <li>acquires an input,</li>
  <li>runs the network,</li>
  <li>computes a loss,</li>
  <li><strong>zeros the network’s parameters’ gradients,</strong></li>
  <li>calls loss.backward() to update the parameters’ gradients,</li>
  <li>calls optimizer.step() to apply the gradients to the parameters.</li>
  </ol>
  <p>Code contains <strong>net.zero_grad()</strong> which has been explained as <strong>zeros the network’s parameters’ gradients</strong>.</p>
  <p>What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase &quot;zeros of networks parameters gradient&quot;. What does that particular step do?</p>
  
Tags: <neural-networks><training><pytorch><gradient>
Title: What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?
UserDisplayName: hanugm
ThreadText: # What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?
  
  (Asked by: username_0 on 2021-07-18)
  
  Consider the [following](https://pytorch.org/docs/stable/notes/modules.html) PyTorch code
  
  ```
  # Run a sample training loop that "teaches" the network
  # to output the constant zero function
  for _ in range(10000):
    input = torch.randn(4)
    output = net(input)
    loss = torch.abs(output)
    net.zero_grad()
    loss.backward()
    optimizer.step()
  
  ```
  
  and its corresponding explanation on training a neural network
  
  A training loop…
  
  1. acquires an input,
  2. runs the network,
  3. computes a loss,
  4. **zeros the network’s parameters’ gradients,**
  5. calls loss.backward() to update the parameters’ gradients,
  6. calls optimizer.step() to apply the gradients to the parameters.
  
  Code contains **net.zero\_grad()** which has been explained as **zeros the network’s parameters’ gradients**.
  
  What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?
  
  **Tags:** neural-networks, training, pytorch, gradient
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-07-18. Score: 2
  
  In the automatic differentiation procedure after **backward** pass
  the gradient with respect to the scalar is added to the current gradient. Without calling **zero\_grad** you will have the sum of all gradients, calcluated before, with the current gradient.
  
  Therefore, `optimizer.step()` will do not this:
  
  ```
  w = w - eta * grad L[i] # L[i] - loss function for the i-th sample
  
  ```
  
  But rather:
  
  ```
  w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples
  
  ```
  
  Which is not the desired behavior.
  
  > **Comments:**
  > **username_2**: Worth noting that this is an implementation detail in PyTorch, and other frameworks may do the same implicitly. It is a nice feature when there's more than one source of gradients, but for simple NNs it's just some boilerplate code that needs to be written by the framework user  
  > **username_0**: So, if we don't use net.`zero_grad()` then the updation of weights will go wrong because of issue in updation of gradients. Updation of weight takes the sum of all the gradients calculated so far instead of the current gradient.  
  > **username_1**: @username_0, yes, precisely, it can be beneficial to use gradient information from the previous timesteps, like in the momentum optimizers - Nesterov, Adam, but not in tihs way. With the decay of the importance of gradients from previous timestamps with time.  
  
  ---
  
  ### Answer by username_3 on 2024-07-09. Score: 0
  
  I agree with the answer from @spiridon\_the\_sun\_rotator .
  
  If you don't zerorize the gradient, before calling loss.backward()
  then gradient will be accumulated which is not what you want. you want to calculate the gradient associated with the current step.
  
  > **Comments:**
  > **username_4**: That answer has already been given.  
  
  ---
  
  