<p dir="auto">Recently I have carefully read the code in <code class="notranslate">pytorch/torch/optim/sgd.py</code> . According to the document in this file, the current implementation of SGD optimizer is as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t - lr*v_(t+1) "><pre class="notranslate"><code class="notranslate">v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t - lr*v_(t+1) 
</code></pre></div>
<p dir="auto">Where <code class="notranslate">v_t</code> is the momentum and <code class="notranslate">w_t</code> is the weight value in step <code class="notranslate">t</code>. <code class="notranslate">grad(x)</code> is the gradient for <code class="notranslate">x</code> .</p>
<p dir="auto">The implementation is OK for this formulation. But I am confused with the implementation of Nesterov Accelerated SGD(NAS). Following the formulation of [1][2], the formulation of NAS under PyTorch's implementation should be</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t + mu*(v_(t+1)-v_t) - lr*v_(t+1) "><pre class="notranslate"><code class="notranslate">v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t + mu*(v_(t+1)-v_t) - lr*v_(t+1) 
</code></pre></div>
<p dir="auto">However, the current implementation of NAS is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t - lr*(grad(w_t) + mu*v_(t+1))"><pre class="notranslate"><code class="notranslate">v_(t+1) = mu*v_t + grad(w_t) 
w_(t+1) = w_t - lr*(grad(w_t) + mu*v_(t+1))
</code></pre></div>
<p dir="auto">I am not sure how to get this formulation. How do you think about this ?</p>
<p dir="auto">[1] <a href="http://cs231n.github.io/neural-networks-3/" rel="nofollow">http://cs231n.github.io/neural-networks-3/</a><br>
[2] <a href="http://www.cs.toronto.edu/~hinton/absps/momentum.pdf" rel="nofollow">http://www.cs.toronto.edu/~hinton/absps/momentum.pdf</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug/Reproduce:</h2>
<p dir="auto">Slicing only sort-of-respects <code class="notranslate">no_grad()</code>. The resulting tensor does not have a history, but does require gradient. However using it does not then materialise a gradient accumulator.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
x = torch.rand(2, 2, requires_grad=True)
with torch.no_grad():
    y = x[:, :1]

z = y.clone()
print(y.requires_grad)  # True
print(z.grad_fn.next_functions)  # ((None, 0),)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-s1">requires_grad</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-k">with</span> <span class="pl-s1">torch</span>.<span class="pl-en">no_grad</span>():
    <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>[:, :<span class="pl-c1">1</span>]

<span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">y</span>.<span class="pl-en">clone</span>()
<span class="pl-en">print</span>(<span class="pl-s1">y</span>.<span class="pl-s1">requires_grad</span>)  <span class="pl-c"># True</span>
<span class="pl-en">print</span>(<span class="pl-s1">z</span>.<span class="pl-s1">grad_fn</span>.<span class="pl-s1">next_functions</span>)  <span class="pl-c"># ((None, 0),)</span></pre></div>
<p dir="auto">For context, this came up during a custom <code class="notranslate">autograd.Function.forward</code> where there's an implicit <code class="notranslate">no_grad()</code>.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">For <code class="notranslate">y</code> not to require gradient.</p>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.6.0</li>
<li>OS (e.g., Linux): Linux</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): conda</li>
<li>Python version: 3.8.3</li>
</ul>