I just figured out this was the problem literally 5 seconds ago when I got a friend to look at the code. The outputs tensor was n X 1 , and the batchY was a size n tensor. So I guess there was undesired broadcasting happening. @ptrblck. Could you suggest ways to account for this in the future? Maybe adding asserts before the loss function  invocation requiring the two inputs to have the same size?
Also another question I had was , why the loss functions in pytorch already don’t do this internally and throw an error to avoid undesired broadcasting? Is there a use for this behavior?
Btw. here is the fit now. (still needs parameter / architecture tuning but it is much better than before)