Alright, tried it  . My confusion matrix starts with only having FNs and TNs and the model never predicts the class I want it to  (still always outputs tensors of 0s)
EDIT: The softmax output seems to be always a tensor of (0.0325, …, 0.0325), already after the first batch