Hello Ereebay!



 Ereebay:

Second, I used the auto.grad function to directly compute the gradient of label, it can track gradients with respect to its arguments normally.


As shown in my post above, in pytorch 0.3.0, F.kl_div() simply
does not track the gradient of its second argument (label).
I downgraded, for whatever reason, to pytorch 0.3.0, so I can’t test
this on later versions.  Could you tell us what version of pytorch you
are using?
(Perhaps someone with pytorch 1.x could run my test on a more
up-to-date version.)
Could you check in a simple script whether or not you can track gradients
with respect to both arguments of F.kl_dv() in your version of pytorch?
(Don’t use your loss-within-a loss code – just make a single call to
F.kl_div() for testing purposes.)


But I compute the gradient of label in the cross entropy loss, it can not track. Is this possible that it can not track an tensor in another loss function ?


I don’t know for a fact, but I would not expect this to be the case.  As far
as I understand it, loss functions are just regular (autograd-supporting)
tensor functions, so I would expect gradient-tracking to work as well
for your loss-within-a-loss use case as it does for the individual loss
functions separately.
Best.
K. Frank