Spot on, thanks. Invoking .retain_grad() on the loss does indeed create a single-element Tensor with value 1. I have some vestige of the leaf concept from graph theory (i.e. no children) in mind, which was messing me up–or it is valid, but my sense of “parenthood direction” is backwards.
Most definitely I am interested in d loss/dw, because it helps me visualize why an optimization might be getting stuck. So how would I resolve that for the loss? Do I actually pull that out of the back-end (i.e. weights), after .backward() is performed?
Excuse my naivete on this, but resolving it will go a long way toward calibrating my intuition. Thanks again.