<p dir="auto">In the cpp backend, undefined Tensors are used to efficiently represent a Tensor full of <code class="notranslate">0</code>s.<br>
Even though we do support this in many functions, there are a lot that almost never get undefined gradients and are thus not properly implemented:</p>
<ul dir="auto">
<li>All the backwards with direct formulas in <code class="notranslate">derivatives.yaml</code></li>
<li>Most functions in <code class="notranslate">Functions.cpp</code>.</li>
</ul>
<p dir="auto">In particular, the assumption that a function with a single output cannot get undefined gradients is not correct.<br>
An example of such issue can be found in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="560858695" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/33037" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/33037/hovercard" href="https://github.com/pytorch/pytorch/issues/33037">#33037</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="555090645" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/32619" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/32619/hovercard" href="https://github.com/pytorch/pytorch/issues/32619">#32619</a></p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ssnl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ssnl">@ssnl</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a></p>
<p dir="auto">In online hard example mining, it is helpful to know the loss for each instance separately.</p>