Hi Mustansar!



 mustansarsaeed:

Hi, Can anyone please guide how can we add the regularizer in the nn.Linear module


Use the weight_decay constructor argument when you instantiate
your optimizer.  For SGD (without momentum), it is the same as
L2 regularization.  For Adam, it is not identical to L2 regularization,
but some people argue that Adam's weight_decay works better
than L2 regularization.  (Conceptually, they both play the same role.)


same like tensorflow.


I don’t know the details of what tensorflow does, so I can’t say how
closely this might mimic the tensorflow implementation.


In Tensorflow we add regularization in the Dense layer as:


If you want to use weight_decay / L2 regularization with just some
layers, you can pass in parameter groups when you instantiate
your optimizer.  Each group can contain different layers and have
different values for weight_decay, including 0.0 (which is its default
value).
Best.
K. Frank