Difficult to tell without the train loop. But seeing from the code, you should use Softmax instead of sigmoid. Sigmoid is for binary classification (it will split the output into 1 if val > 0.5 and 0 otherwise).