Ah, I didn’t know this about Adam.
I also tried a higher learning rate, but if it is higher than 1e-8 for SGD, it explodes. However, if it is this low, it could learn ages. I tried momentum as well. Actually, I can not think of something I haven’t tried yet.
The X_train is a kernel matrix and this code does the same as LinearRegression:
import numpy as np
import numpy.linalg as la
from sklearn.metrics import mean_absolute_error

alpha = np.dot(la.inv(X_train), y_train)
y_predicted3 = np.dot(X_test, alpha)
error3 = mean_absolute_error(y_test, y_predicted3)
print(error3)

There is no regularization. I tried to optimize L1, but as it is convex problem, I thought it would converge to the same result. I could have added a commented line with L1Loss, I tried it, but it did not converge to the analytical solution as well. I compare L1 errors because it is easier to see the physical meaning of the problem.