Option 4 ( 2 output neurons and [raw logits + nn.CrossEntropyLoss ] ) works!
Although I need several thousand epochs on the whole batch [23445, 128]
Without normalization.
SGD, lr=0.1, momentum=0.9
I will try different hyper-parameters.
Result
# Loss
tensor(0.1288, device='cuda:0')
# Output
tensor([[ 3.0039e+00, -3.0803e+00],
        [ 1.6767e+00, -1.9099e+00],
        [-1.0335e+00,  9.7911e-01],
        ...,
        [-9.0753e-01,  7.0044e-01],
        [-4.2910e-01,  2.0413e-01],
        [-9.9083e-01,  7.8327e-01]], device='cuda:0')
# Target
tensor([ 0,  0,  1,  ...,  1,  1,  1], device='cuda:0')