The parameters of a model should be consistency, no matter what mode it is (train or eval). So your patch commit is the right way to go.

Yes, that is correct. I thought only the hidden layers were needed but actually one needs to recreate the whole model.

And the parameters you used when training and inference have different values, so it will have different output. I do not know what you are actually doing here but the output of training and inference don’t match seems to reasonable to me.

Not at all in this case. Let me elaborate more. Those outputs shown above come from predicting over the training set using the state_dict of the epoch that fulfilled the criterion for training. That is the state_dict I saved so the predictions they have to match. Now, I found that the error was that I did not scale the features with the same scaler used for training.

And, it’s weird to loop the parameters in  forward  , you can refactor your code like this:
slope = getattr(slope_name, self)
intercept = getattr(intercept_name, self)


Thanks, @chenglu  I will do that.