I just checked my commits and I was using  torch.stack but dropped it at some point. With that change, now I am getting this (thanks for spotting that!);
outputs
tensor([[-14.5754413605],
        [-14.5754413605]], grad_fn=<StackBackward>)
targets
tensor([-14.5868730545, -14.5640010834])
Diff in intercept_Cu
Diff in slope_Cu
Diff in linears.Cu.0.weight
Diff in linears.Cu.0.bias
Diff in linears.Cu.2.weight
Diff in linears.Cu.2.bias
Diff in linears.Cu.4.weight
Diff in linears.Cu.4.bias

Optimized parameters for Cu symbol
Index 0
Parameter containing:
tensor([[-3.2861328236e-06,  2.6473844628e-05,  3.5650995045e-08,
          1.4864326658e-06, -1.0720622959e-04,  3.0406141605e-08,
          1.3593539488e-11, -1.9859398570e-09],
        [-5.8556226534e-10, -1.1646034137e-10, -3.3600910682e-12,
          6.7106892265e-10,  1.8430428624e-13,  1.9599950107e-08,
          2.4708364435e-05,  3.2043487863e-07],
        [ 1.9733222143e-04,  1.3596744564e-10,  1.6436105810e-08,
         -3.0531379647e-09, -6.3785437305e-06,  7.4555270811e-13,
          8.1752958067e-05, -1.1181630725e-05],
        [-1.2528111881e-09,  4.6580535127e-06,  2.3549859979e-12,
          2.8091712984e-11,  1.7993905931e-04, -1.4886735508e-12,
         -9.2567507479e-12,  2.8859590202e-06],
        [ 3.4168685943e-06, -5.6807679357e-05,  1.2368669559e-05,
         -2.1798576399e-12, -1.4500128600e-05, -3.1362407071e-07,
          1.7807322283e-09,  9.7959136838e-06],
        [ 8.9927290503e-08, -5.2266013739e-09, -9.1957379753e-14,
         -2.9821003977e-07, -1.7513568764e-06,  1.0372443600e-13,
          1.2319574694e-07, -3.6574114347e-05],
        [ 5.3554540500e-06, -1.8524660845e-05,  1.1853338037e-05,
         -2.1492420638e-04,  2.3621556466e-05, -8.0939061009e-11,
         -6.9240194023e-08,  1.5314364646e-05],
        [-2.0754782781e-08, -1.9774879547e-05, -1.3601642422e-05,
          5.2368657634e-05,  3.3497635741e-05, -5.8766081565e-06,
         -6.5623047703e-05,  3.9108752389e-05],
        [-1.5761332861e-06, -7.3087621786e-06, -2.9493878628e-07,
         -4.5463502829e-07, -3.2682427786e-07, -1.4819252101e-05,
          2.6041425372e-05, -1.0358776308e-06],
        [-6.6475655558e-07, -1.3438479496e-10,  1.8068027430e-07,
         -2.5042306007e-09, -5.2879945542e-06, -6.9557786446e-06,
          2.9763690179e-08, -2.8302894425e-14]], requires_grad=True)
Gradient tensor(-6.9187924964e-05)

Index 1
Parameter containing:
tensor([ 0.0316366255,  0.2617242932,  0.3015798032, -0.0021922502,
        -0.0615932010, -0.2602513433, -0.0311477333, -0.2361671627,
         0.1662444025, -0.0660640150], requires_grad=True)
Gradient tensor(2.4240243301e-16)

Index 2
Parameter containing:
tensor([[ 9.7365699503e-08,  4.2661922635e-06,  1.9631031591e-06,
         -4.2697833123e-05,  6.2208728195e-06,  1.1655485604e-12,
         -6.4603467763e-05, -1.3209832117e-09,  1.1756450391e-07,
         -7.9867913882e-07],
        [ 6.2096376041e-07, -6.1568898673e-06, -7.3711348136e-10,
          3.8170369088e-09, -1.4567660855e-06, -1.9914123186e-06,
          2.3581033020e-06, -1.4400919781e-06,  1.2830110308e-09,
         -9.6331113753e-08],
        [-1.5564822888e-06,  2.5510303203e-06,  1.5743958670e-07,
         -1.4166996607e-06, -1.3405845323e-07, -9.9951203083e-06,
          1.0537170965e-05,  7.7086369856e-06,  2.7015998813e-11,
          4.8960191457e-10],
        [-1.7156789909e-06, -2.3564821277e-06,  1.5615292262e-13,
          3.7418202217e-13, -4.3808788178e-05, -1.6505031454e-05,
         -8.4225684986e-06,  4.8483889259e-06, -5.0767212656e-08,
          2.3069132737e-07],
        [ 2.8512771678e-05, -7.5202997323e-06,  1.7333918549e-06,
          1.2562820473e-07,  5.2780844271e-05, -1.1339360562e-09,
          3.8166854659e-13, -1.2022780993e-07,  7.6206299127e-05,
          1.7692066194e-06],
        [-1.8082403130e-07,  1.8067876226e-06,  1.5731624337e-10,
         -3.9476603410e-12, -1.6683844706e-07,  1.6850806333e-06,
         -2.5483970489e-10, -1.8325088604e-05, -3.7972899918e-06,
          1.8083280651e-08],
        [ 1.2292147552e-12,  9.3183753052e-06, -1.0420450280e-07,
          3.0822411645e-07,  6.6431852019e-07,  9.7349062145e-10,
          3.3600823372e-05, -2.3434172908e-04, -4.7051515462e-11,
          3.7607719605e-08],
        [ 2.9142852020e-10,  1.6880323983e-07,  3.4797506032e-06,
          2.2227823138e-08, -8.4504938513e-07, -1.0985943663e-04,
          2.5039498723e-05,  8.6511966211e-13,  1.6281462740e-04,
         -2.8856720746e-07],
        [-2.9593747968e-06, -5.8458951457e-08, -5.6971380502e-08,
         -1.2519759184e-04, -7.4558295735e-13, -2.9341401842e-07,
          4.2668673927e-08,  4.3059226300e-06, -2.8244965478e-08,
         -5.0044291129e-06],
        [ 3.0511776004e-06, -1.0126113713e-15,  5.8472587625e-06,
         -7.7287486420e-06, -1.2084972241e-06, -4.0337028162e-10,
          7.3831834015e-05,  1.5755430240e-06,  1.3774927379e-12,
          1.3072969159e-04]], requires_grad=True)
Gradient tensor(-0.0001017772)

Index 3
Parameter containing:
tensor([ 0.1047996879, -0.2729528248,  0.2065724581,  0.0899153724,
        -0.1029036865, -0.3049356043, -0.2874263823,  0.1054596379,
        -0.2320392430, -0.0505143851], requires_grad=True)
Gradient tensor(-1.4201264947e-12)

Index 4
Parameter containing:
tensor([[ 1.3515332284e-06,  1.9205540269e-08,  1.9853887352e-05,
          2.2419371817e-06,  1.0494016323e-08, -4.0517086745e-05,
          1.4675807324e-04,  5.4397496285e-08, -6.9757788879e-08,
          3.3906806038e-06]], requires_grad=True)
Gradient tensor(0.0001229179)

Index 5
Parameter containing:
tensor([-0.2859740555], requires_grad=True)
Gradient tensor(-7.3175310256e-09)


The two variables that were not changing before now they are changing. However, outputs remained the same even though the loss function value at epoch 1000 was 1.643620e-05. Do you know any idea why? Doesn’t that mean that outputs are very near to the targets? Meanwhile, I am playing with learning rate and weight decay and see what happens.