Thank you @ptrblck , How can I apply reularization using LBFGS optimizer as if I understand correctly the Weight decay term is the regularization term in Pytorch but LBFGS doesnt have a weight decay term.