Nice title!
I am not sure to perfectly understand what you want, but you can avoid regularization of your parameters when you define the optimizer by passing weight_decay=0:
torch.optim.Adam(not_regularized_params, lr=0.001, weight_decay=0)

However, I think it’s the default parameter for most of optimizers.