L1Loss and L2Loss are both convex, but they are two different functions, so optimizing one or the other will give different results !
For example, considering a 1D problem without bias, if your samples are (input, output): (1, 0.5) and (2, 0). Then your loss function will look like this:

image.png946×438 29.6 KB

As you can see, the absolute value and the squared norm don’t have the minimum at the same point
It is surprising that you cannot use SGD with any value bigger than 1e-8. Maybe you want to normalize your dataset to avoid very bad conditioning that significantly impact simple gradient based methods.