I tried scaling my inputs between -1 and 1, and then dividing the loss by a bit, (although it is still in the 6 digits). I still see gradient explosions, later in the neural network. Should I try to future scale down the loss and should I try gradient clipping if that doesn’t work?
Example:
coordReconstruction_layer.0.10.bias tensor([  29565.5938,  107157.0859, -415445.0000,  -52302.0312, -174125.9688,
          91306.4375,  178269.4062, -365361.7500,  573210.0000,  207930.5625,
         -18185.3281,   58567.1094, -360612.0625,  128182.8359,   29353.1719,
        -199463.4062,  -17777.7500, -110101.8203, -532024.7500,  160549.6250,
        -576055.6875,  189424.9844,  157812.2344, -245538.3906,   85801.1406,
         176016.4375,  224826.5312,   23766.4766, -619454.1875, -294117.7500,
        -325290.5312,   60833.8672, -107854.8438,  -57861.6602, -264200.2500,
          83863.8906,  172188.1406,  385307.7500, -346806.8125,  108188.7266,
         440669.6250,  -97171.9609, -602740.8125,  129569.0156, -736367.2500,
          20825.3438,  -35092.2891,   22432.6875, -297975.8438, -282776.5938,
         226446.8125,   77869.7969,  -57261.6875, -323141.1875,  362888.2500,
        -118788.5781,   48330.5234, -223928.7188,  507906.5000,  -31208.7812,
        -106249.6562,  274496.8750, -547059.8750,  354355.5000],
       device='cuda:0')
coordReconstruction_layer.1.0.weight tensor([[[[ 1.2466e+06]],

         [[ 1.6526e+06]],

         [[ 5.6299e+06]],

         [[ 3.0220e+06]],