Thanks to dear God, I finally found the culprit! it was/is the BCE!
For some weird reason the former versions of Pytorch worked perfectly fine with reduction='mean', however, in the newer Pytorch versions that I tested myself , including (1.1.0 and ultimately 1.2.0+cu92) only reduction='sum' will work without a hitch!
I do not have any idea what happened in the course of updating Pytorch 0.4 to 1.1.0 that results in such weird behavior.