Solved by Shisho_Sama in post #20 
              
              
                Thanks to dear God, I finally found the culprit! it was/is the BCE! 
For some weird reason the former versions of Pytorch worked perfectly fine with reduction='mean', however, in the newer Pytorch versions that I tested myself , including (1.1.0 and ultimately 1.2.0+cu92) only reduction='sum' will w…