Simple L2 regularization?
Hi, does simple L2 / L1 regularization exist in pyTorch? I did not see anything like that in the losses. I guess the way we could do it is simply have the data_loss + reg_loss computed, (I guess using nn.MSEloss for the L2), but is an explicit way we can use it without doing it this way? Thanks
Kalamaya
Hi, <SCODE>l1_crit = nn.L1Loss(size_average=False)
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
loss += factor * reg_loss
<ECODE> Note that this might not be the best way of enforcing sparsity on the model though.
fmassa
Kalamaya
Yeah, that’s been added there as an optimization, as L2 regularization is often used.
apaszke
Kalamaya
what do you recommend which would be a better way to enforce sparsity instead of L1?
ecolss
fmassa
ecolss
example: <SCODE>xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx)
<ECODE>
ncullen93
<SCODE>xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
target = Variable(torch.from_numpy(np.zeros((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx, target)
<ECODE>
fmassa
I have two questions about L1 regularization: 
How do we backpropagate for L1 regularization? I wonder it because the term is not differentiable.
 
Where can I see the implementation of L1 regularization? In the following link, there is only pass.

Ja-Keoung_Koo
smth
gwg
If you add the L1 regularization to the loss as I explained in a previous post, the gradients will be handled by autograd automatically.
fmassa
yibo
<SCODE>batch_loss = loss(input=y_pred,target=batch_ys)
batch_loss += lambda*R
<ECODE> ?
Brando_Miranda
hughperkins
Neta_Zmora
<SCODE>weight_p, bias_p = [],[]
for name, p in model.named_parameters():
  if 'bias' in name:
    bias_p += [p]
  else:
    weight_p += [p]

optim.SGD(
  [
    {'params': weight_p, 'weight_decay':1e -5},
    {'params': bias_p, 'weight_decay':0}
  ],
  lr=1e-2, momentum=0.9
)
<ECODE>
shirui-japina
We do regularization to handle the high variance problem (overfitting). It’s good if the regularization includes all the learnable parameters (both weight and bias). But since bias is only a single parameter out of the large number of parameters, it’s usually not included in the regularization; and exclusion of bias hardly affects the results.
Deepak