Ok, but anyway LinearRegression is using L2 norm and I used it for the Linear layer too. So the conditions for both are the same and as such, they should have the same errors in the end.
Even if I normalize the data, LBFGS has a different result. The maximum learning rate for SGD is in order 1e-3 and it could again learn for ages.