Depends which criterion and which regularization you are using.
In pytorch, weight decay is usually (for sgd / adam) included in the optimizer directly and not the loss. This is because this term does not need to be computed explicitly.