Thanks Vahid for replying. So just for the reference let me mention that I am doing a time series forecast. I use LBFGS optimizer here.
I am just trying to reduce the degrees of freedom/variables, so when I reproduced, I noticed it is not really dependent on the loading different models, although it happens with different models as well; so I am loading and testing a single model 3 times below.
Single load and test run contains print statements in the form:-
<<< test loss
error
i/p tensor
o/p or forecast tensor>>>
Output on console:
test loss: 3.921948823121292
Weighted mean absolute error is : 3.519775891348272
input tensor([[ 31.0200,  31.0400,  31.0500,  31.0800,  31.0900,  31.1200,
          31.1400,  31.1600,  31.1800,  31.2000,  31.2300,  31.2500,
          31.2700,  31.2900,  31.3100,  31.3300,  31.3500,  31.3700,
          31.3900,  31.5400,  31.5600,  31.6900,  31.7100,  31.7300,
          31.7600,  31.7800,  31.7900,  31.8200,  31.8400,  31.8600,
          31.8900,  31.9000,  31.9200,  31.9400,  31.9600,  31.9800,
          32.0000,  32.0300,  32.0600,  32.0900]], dtype=torch.float64, device='cuda:0')
forecast tensor([ 41.3981,  35.4583,  30.2360,  28.8500,  29.3385,  30.0793,
         30.4554,  30.4568,  30.2663,  30.0545,  29.9258,  29.9056,
         29.9869,  30.1447,  30.3500,  30.5775,  30.8080,  31.0286,
         31.2320,  31.4916,  31.6959,  31.9196,  32.0829,  32.2035,
         32.3037,  32.3846,  32.4457,  32.5049,  32.5549,  32.5969,
         32.6384,  32.6662,  32.6908,  32.7136,  32.7347,  32.7543,
         32.7726,  32.7959,  32.8216,  32.8481], dtype=torch.float64, device='cuda:0')
test loss: 575158.8622441115
Weighted mean absolute error is : 95.97287345358812
input tensor([[ 31.0200,  31.0400,  31.0500,  31.0800,  31.0900,  31.1200,
          31.1400,  31.1600,  31.1800,  31.2000,  31.2300,  31.2500,
          31.2700,  31.2900,  31.3100,  31.3300,  31.3500,  31.3700,
          31.3900,  31.5400,  31.5600,  31.6900,  31.7100,  31.7300,
          31.7600,  31.7800,  31.7900,  31.8200,  31.8400,  31.8600,
          31.8900,  31.9000,  31.9200,  31.9400,  31.9600,  31.9800,
          32.0000,  32.0300,  32.0600,  32.0900]], dtype=torch.float64, device='cuda:0')
forecast tensor([  705.8100,   972.7171,  1061.4445,  1062.1935,  1023.6047,
          972.3282,   921.9674,   878.4381,   843.3888,   816.3613,
          796.0533,   780.9891,   769.8488,   761.5599,   755.3051,
          750.4875,   746.6833,   743.5984,   741.0307,   738.8185,
          736.7482,   734.8179,   732.9316,   731.1557,   729.5226,
          728.0333,   726.6906,   725.4937,   724.4122,   723.4327,
          722.5403,   721.7137,   720.9582,   720.2620,   719.6152,
          719.0100,   718.4400,   717.8991,   717.3720,   716.8506], dtype=torch.float64, device='cuda:0')
test loss: 45029.746996055845
Weighted mean absolute error is : 87.00255914868971
input tensor([[ 31.0200,  31.0400,  31.0500,  31.0800,  31.0900,  31.1200,
          31.1400,  31.1600,  31.1800,  31.2000,  31.2300,  31.2500,
          31.2700,  31.2900,  31.3100,  31.3300,  31.3500,  31.3700,
          31.3900,  31.5400,  31.5600,  31.6900,  31.7100,  31.7300,
          31.7600,  31.7800,  31.7900,  31.8200,  31.8400,  31.8600,
          31.8900,  31.9000,  31.9200,  31.9400,  31.9600,  31.9800,
          32.0000,  32.0300,  32.0600,  32.0900]], dtype=torch.float64, device='cuda:0')
forecast tensor([ 167.3744,  209.8192,  215.1611,  211.6978,  209.1109,  209.4348,
         212.1701,  216.3190,  221.0366,  225.7625,  230.1856,  234.1582,
         237.6411,  240.6542,  243.2452,  245.4709,  247.3874,  249.0455,
         250.4888,  251.7960,  252.9559,  254.0158,  254.9618,  255.8026,
         256.5561,  257.2335,  257.8430,  258.4001,  258.9106,  259.3803,
         259.8172,  260.2196,  260.5931,  260.9413,  261.2670,  261.5722,
         261.8588,  262.1313,  262.3911,  262.6389], dtype=torch.float64, device='cuda:0')

Process finished with exit code 0