One way is to use optimizer on grouped parameter. If optimizer won’t update your weights the loss don’t have any affect. In that way you can keep the same loss function for all parameters and just filter in the optimizer.