Hello Capo!
The short answer is that you have to rewrite your loss function to
properly use autograd so that loss-function gradients can flow back
through your model and can be used to update your model’s parameters
(The most straightforward way to do this is to implement the entirety of
your loss-function computation with pytorch tensor operations, in which
case you get the autograd machinery “for free.”)
[quote=“Capo_Mestre, post:1, topic:95790, full:true”]
# mean squared error with explicit const and linear terms
def MSE_toOptimize(params,yHat,y):
    ...
    x = torch.tensor(x).to('cuda:0')
    ...
    return res_.cpu().detach().numpy()

def optimizedParametersAre(yHat,y):
    ...
    return torch.tensor(params).to('cuda:0')

def MSE_mod(yHat,y):
    ...
    loss__ = torch.tensor(loss__).to('cuda:0').requires_grad_(True)
    return loss__

If you are indeed using the above loss function for the loss you
backpropagate by calling loss.backward(), the backpropagation
won’t work.
The various .to('cuda:0') calls “break the computation graph”
(unless the tensors in question are already on 'cuda:0', in which
case they’re a no-op), and .detach().numpy() in any event breaks
(that part of) the computation graph.
Lastly, calling .requires_grad_(True) in:
loss__ = torch.tensor(loss__).to('cuda:0').requires_grad_(True)

doesn’t fix the problem.  It gives you a nice, new pytorch tensor that
will track its gradient going forward, but the damage has already
been done.


The question is why the Loss-values within an epoch repeat each other from one epoch to another without showing any evolution over epochs?


As a consequence of the above, non-trivial gradients never flow back
to your model parameters, so your call to optimizer.step() (which
you don’t show, but I’m guessing you have) doesn’t actually update
your model – your model remains unchanged throughout your training
run.
You don’t show how you prepare your batches, but if you don’t shuffle,
or otherwise randomize your training data when you create the batches,
then all the epochs will be cycling through the same set of batch with
the same model, hence giving you the same loss values over and over
again.
(But because, within an epoch, one batch is different from the next,
your loss function will vary within an epoch, but repeat from epoch
to epoch.)
Good luck.
K. Frank