Solved by ptrblck in post #4 
              
              
                No, you would have to add all layers which should be regularized to the loss calculation or to the optimizer’s parameter list containing the weight_decay.