policy is the weight of loss.grad, not the weight of loss itself.

taken as a scalar quantity (that’s what I mean by weight) it’s just the same: grad(w*x) = w*grad(x)
you just have to make sure you are not using it as a variable of the tree (using pi.detach() should do it)