Great ! I could copy labels[:,0] to mult to make it not requiring the gradient. Thanks
Finally I did this:
    loss_parameters = F.smooth_l1_loss(outputs, labels, reduction='none')
    loss_parameters[:, 1, :, :] *= labels[:, 0, :, :]

It’s slightly different, but since it’s a L1 norm, it’s almost the same.