Thanks a lot @KFrank  for a great reply! Everything seems to make sense.



 KFrank:

As a consequence of the above, non-trivial gradients never flow back
to your model parameters, so your call to optimizer.step() (which
you don’t show, but I’m guessing you have) doesn’t actually update
your model – your model remains unchanged throughout your training
run.


Yes, I do use this command in the training process.



 KFrank:

You don’t show how you prepare your batches, but if you don’t shuffle,
or otherwise randomize your training data when you create the batches,
then all the epochs will be cycling through the same set of batch with
the same model, hence giving you the same loss values over and over
again.


Indeed, I don’t shuffle the training data.
I will try to think how to rewrite the code, so that my custom loss-function would work properly having in mind the understanding that your reply brought.



 KFrank:

and .detach().numpy() in any event breaks
(that part of) the computation graph.


The reason why I did detach().numpy() is that the function minimize was not happy about the torch.tensor() data format as the output of the function MSE_toOptimize.