If you’re using scikit-learn, perhaps try using skorch? It’s a PyTorch version of scikit-learn that wraps around it. (github repo: GitHub - skorch-dev/skorch: A scikit-learn compatible neural network library that wraps PyTorch 1)
The main issue is that the outputs of your model are being detached, so they have no connection to your model weights, and therefore as your loss is dependent on output and x (both of which are detached), your loss will have no gradient with respect to your model parameters! Which is why it’s not decreasing!
You’ll need to calculate your loss value without using the detach() method at all. Also, you don’t need the loss = Variable(loss, requires_grad=True) line, I think!