<p dir="auto">Hi,<br>
in many applications will be useful to have available not only the gradients averaged on minibatches, but also its second momentum for each parameter.<br>
I think developers can do it with a small computation overhead by modifying accumulate_grad.cpp (I was not able to understand the underlying class structure), accumulating also the square of the individual contribution to the gradient (which is much more simple that getting the individual contribution to the gradients as requested in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="325590018" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/7786" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/7786/hovercard" href="https://github.com/pytorch/pytorch/issues/7786">#7786</a>).<br>
I'm aware that it will duplicate the memory requirements, but hopefully it can be done as optional.<br>
Thank you very much.</p>
<p dir="auto">If you have a question or would like help and support, please ask at our<br>
<a href="https://discuss.pytorch.org/" rel="nofollow">forums</a>.</p>
<p dir="auto">If you are submitting a feature request, please preface the title with [feature request].<br>
If you are submitting a bug report, please fill in the following details.</p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">Provide a short description.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">Please try to provide a minimal example to repro the bug.<br>
Error messages and stack traces are also helpful.</p>
<h2 dir="auto">System Info</h2>
<p dir="auto">Please copy and paste the output from our<br>
<a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="nofollow">environment collection script</a><br>
(or fill out the checklist below manually).</p>
<p dir="auto">You can get the script and run it with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py"><pre class="notranslate"><code class="notranslate">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre></div>
<ul dir="auto">
<li>PyTorch or Caffe2:</li>
<li>How you installed PyTorch (conda, pip, source):</li>
<li>Build command you used (if compiling from source):</li>
<li>OS:</li>
<li>PyTorch version:</li>
<li>Python version:</li>
<li>CUDA/cuDNN version:</li>
<li>GPU models and configuration:</li>
<li>GCC version (if compiling from source):</li>
<li>CMake version:</li>
<li>Versions of any other relevant libraries:</li>
</ul>
<p dir="auto">Currently, loading checkpoints using <code class="notranslate">strict=False</code> can be fairly error-prone. An extra level (for example with a <code class="notranslate">nn.DataParallel</code>) could lead to completely missing loading all the weights, and there is no sign that this has happened.</p>
<h3 dir="auto">Proposal:</h3>
<p dir="auto">A way to enable user-code to verify for possible mistakes would be to add an option to <code class="notranslate">load_state_dict</code> to return matched / unmatched / unexpected keys.<br>
This would allow the user to easily inspect if the loading is actually doing what is expected.</p>
<p dir="auto">Returning those values is only needed if <code class="notranslate">strict=False</code>, because in <code class="notranslate">strict=True</code> mode an error is raised if the state dict doesn't match exactly.</p>
<p dir="auto">A possible API:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def load_state_dict(self, state_dict, strict=True, return_unmatched=False):
    ...
    if return_unmatched:
        return matched, unmatched, unexpected"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">load_state_dict</span>(<span class="pl-s1">self</span>, <span class="pl-s1">state_dict</span>, <span class="pl-s1">strict</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">return_unmatched</span><span class="pl-c1">=</span><span class="pl-c1">False</span>):
    ...
    <span class="pl-k">if</span> <span class="pl-s1">return_unmatched</span>:
        <span class="pl-k">return</span> <span class="pl-s1">matched</span>, <span class="pl-s1">unmatched</span>, <span class="pl-s1">unexpected</span></pre></div>
<p dir="auto">Thoughts?</p>