You shouldn’t pass the softmax into the CrossEntropy loss. It computes log_softmax(y2) internally, so you end up with with log_softmax(softmax(z)), which would make for a pretty awkward gradient. That was actually frequent issue among my students so I made a kidn of cheatsheet for them: https://sebastianraschka.com/faq/docs/pytorch-crossentropy.html 293

@ptrblck, I did try removing the softmax as well. In this case the error decreases but the accuracy does not improve.

That’s good. Now that the loss decreases, the next step is to find out why the accuracy doesn’t increase. Can you show the function that computes the accuracy?